[Tile-serving] [openstreetmap/osm2pgsql] Save unprojected coordinates in node caches and use osmium dense file array (#668)

[Sarah Hoffmann]

(not ready yet, I want CI to check compiling on Windows and Mac)

*This is a DB-breaking change.* DBs imported with older code can no longer be updated with this code and there will be **no migration** to the new format. 

Unprojected coordinates in node caches
-------------------------------------------------------

Reprojection of coordinates into the tile coordinate system is delayed until the point when nodes are retrieved from the middle caches (see in particular the `nodes_get_list()` functions). This has a few advantages:

 * OSM coordinates can be saved directly, no need for fix-point magic on reprojected coordinates, which avoids a few rounding errors.
 * no more custom scale factor needed for custom projections (fixes #121)
 * no need for floating-point implementation of caches (supersedes #357) 
 * avoid expensive floating point conversation until last minute

The main drawback is that some coordinates now may need to be reprojected multiple times. This is for example the case for nodes where roads cross each other. OTOH unused nodes are no longer converted.

Osmium Dense File Array
-----------------------------------

The custom flatnode file implementation has been replaces with osmium's dense file array index. Its implementation uses mmap, avoiding custom block management. You can also use osmium-based tools to inspect/update the file outside osm2pgsql.

Furthermore, the dense file array should be thread-save for reading. The 4th commit in this PR removes the parts where the file is reopened and uses one flatnode store object throughout the import.  (fixes #96)


There are a couple of additional formatting changes created by git-clang-format.

Resolves #542.
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/osm2pgsql/pull/668

-- Commit Summary --

  * store unprojected coordinates in node caches
  * remove scale option
  * replace flatnode store with osmium dense file array
  * allow concurrent reads on flatnode file

-- File Changes --

    M geometry-processor.cpp (12)
    M geometry-processor.hpp (7)
    M middle-pgsql.cpp (125)
    M middle-pgsql.hpp (10)
    M middle-ram.cpp (37)
    M middle-ram.hpp (5)
    M middle.hpp (11)
    M node-persistent-cache.cpp (654)
    M node-persistent-cache.hpp (101)
    M node-ram-cache.cpp (356)
    M node-ram-cache.hpp (81)
    M options.cpp (44)
    M options.hpp (4)
    M osmdata.cpp (20)
    M output-gazetteer.cpp (9)
    M output-gazetteer.hpp (11)
    M output-multi.cpp (106)
    M output-multi.hpp (7)
    M output-null.cpp (8)
    M output-null.hpp (4)
    M output-pgsql.cpp (25)
    M output-pgsql.hpp (4)
    M output.hpp (4)
    M tests/middle-tests.cpp (17)
    M tests/mockups.hpp (16)
    M tests/regression-test.py (36)
    M tests/test-middle-flat.cpp (1)
    M tests/test-middle-pgsql.cpp (1)
    M tests/test-middle-ram.cpp (1)
    M tests/test-output-pgsql-area.cpp (1)
    M tests/test-output-pgsql.cpp (7)
    M tests/test-parse-diff.cpp (6)
    M tests/test-parse-xml2.cpp (5)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/668.patch
https://github.com/openstreetmap/osm2pgsql/pull/668.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/668
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20170104/c11d16ba/attachment.html>