[Tile-serving] [osm2pgsql] Make more of osm2pgsql threading aware (#110)

[apmon]

This is the pull request for the "threading" branch. This patch makes the initial import of ways and relations multi-threaded. Given that on a fast machine (with SSDs) most of those phases can be CPU bound, moving things into a multi threading approach can speed these phases up significantly.
You can merge this Pull Request by running:

  git pull https://github.com/apmon/osm2pgsql threading3

Or you can view, comment on it, or merge it online at:

  https://github.com/openstreetmap/osm2pgsql/pull/110

-- Commit Summary --

  * Fix C based tagtransform for non-hstore mode
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * possix_fallocate does not set errno (fixes #55)
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * If the tablespacetest already exist use that instead of requiring sudo permissions to create one
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Improve tag copying of outer way tags to multi-polygon relations
  * improve reporting if osm2pgsql fails to execute in tests
  * Enable hstore tests and --extra-attributes test
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Add more tests for multipolygon diff processing.
  * Merge branch 'master' of git://github.com/openstreetmap/osm2pgsql
  * Fix errors if not all nodes fit into cache
  * Add a (broken, disabled) test for non 100% node cache hit ratio
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Improve style.lua to pass more of the multi-polygon tests
  * The pbf reader was not adding osm_uid and osm_user with -x
  * Fix a segfault when cache is set to 0.
  * Round results rather than truncate the in the DOUBLE_TO_FIX macro
  * Add some basic tag counting to check things don't change
  * Improve configure warning for missing lua dependencies
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Add the parameter --tag-transform-script to -h -v output of osm2pgsql
  * Add tests for multi-polygon diff import issues reported in trac #2853
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Another attempt at improving the copying of multi-polygon tags from outer ways
  * Fixed up multi-polygon tag handling
  * Fix summery statistics in test after the change of stylesheet caused the tests to fail
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Merge branch 'master' of https://github.com/openstreetmap/osm2pgsql
  * Warn and exit if both -j and -k are passed as these are exclusive options
  * Use worker threads to speedup output processing of way and relations import
  * Move more out of the feeder threads and into the worker threads.
  * Limit threading to 6 threads for now, as things don't scale well beyond that
  * Run multi-polygon tests with multi-threading as well.
  * Fix memory read after free bug found via valgrind
  * The strategy for organising the db by geographic order was flawed

-- File Changes --

    M build_geometry.cpp (88)
    M build_geometry.h (12)
    M expire-tiles.c (1)
    M keyvals.c (68)
    M middle-pgsql.c (409)
    M middle-ram.c (53)
    M middle.h (35)
    M node-persistent-cache-reader.c (53)
    M node-persistent-cache.c (283)
    M node-persistent-cache.h (17)
    M node-ram-cache.c (2)
    M node-ram-cache.h (2)
    M osm2pgsql.c (30)
    M output-gazetteer.c (39)
    M output-pgsql.c (1015)
    M output-pgsql.h (34)
    M output.h (6)
    M parse-pbf.c (2)
    M tagtransform.c (72)
    M tagtransform.h (12)
    M tests/regression-test.py (4)
    M text-tree.c (46)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/110.patch
https://github.com/openstreetmap/osm2pgsql/pull/110.diff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20131212/d3e5f988/attachment.html>