[Tile-serving] [openstreetmap/osm2pgsql] New ram middle (#1461)

[Jochen Topf]

This implements a completely new ram (non-slim) middle that is compatible with the old one but much more memory-efficient. The first three commits build some infrastructure for it, the last contains the new middle.

The new middle usually only stores node locations and way nodes which are needed to build way and relation geometries. If the output uses two-stage processing it can now tell the middle that that's the case and then also complete way objects (including the tags and attributes) are stored. The new middle code has provisions for storing node and relation objects, too, but these are not used yet, because two-stage processing does not use them yet.

When not using two-stage processing the memory requirements are much much smaller than with the old ram middle. Rule of thumb is, you'll need about 1GB plus 2.5 times the size of the PBF file as memory. This makes it possible to import even continent-sized data on reasonably-sized machines.

When using two-stage processing the memory requirements are larger than without it. Currently OSM objects are just stored in the libosmium internal format, which is meant for quick access, not for saving memory. There could be considerable space savings there with a better implementation, but considering that two-stage processing is seldom used and you can always fall back to the slim middle, improving this has been left for a later time.
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/osm2pgsql/pull/1461

-- Commit Summary --

  * Add infrastructure for asking outputs for their requirements
  * Add new ordered_index_t class
  * New node_locations_t class for storing node locations in memory
  * New ram middle that's much more memory-efficient

-- File Changes --

    M src/CMakeLists.txt (2)
    M src/middle-ram.cpp (327)
    M src/middle-ram.hpp (188)
    M src/middle.hpp (3)
    A src/node-locations.cpp (117)
    A src/node-locations.hpp (94)
    M src/options.hpp (7)
    A src/ordered-index.cpp (78)
    A src/ordered-index.hpp (183)
    M src/osm2pgsql.cpp (2)
    M src/output-flex.cpp (7)
    M src/output.hpp (6)
    M tests/CMakeLists.txt (2)
    M tests/common-import.hpp (4)
    M tests/test-middle.cpp (40)
    A tests/test-node-locations.cpp (82)
    A tests/test-ordered-index.cpp (146)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/1461.patch
https://github.com/openstreetmap/osm2pgsql/pull/1461.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/1461
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210426/15681407/attachment.htm>