[Tile-serving] [osm2pgsql-dev/osm2pgsql] Daylight Ingest and Slow Deletions (Discussion #2088)

[mboeringa]

I now noticed in the osmium manual, that the Meta suggested `apply-changes` option of osmium, will sort the input files, meaning the entire pbf file needs to be read in memory, which according to the same manual page will likely cost about 10x the amount of RAM as the size of the input compressed file.

_No wonder I ran out of RAM_, that means at least 1TB(!) of RAM needed for the 99GB gzipped changeset file... The suggestion by Meta seems wrong with the current new gigantic sidecar file with 1.2B buildings, few if any people in the OSM community will have a server with that much RAM. Even for the Cloud providers, an instance with that much RAM is rare.

Don't no if it will work, but will now try to:
- Convert the osm.gz file to osm.pbf using the osmium `cat` command, which so far uses negligible memory.
- Merge the two pbfs (Planet + building sidecar) using osmium `merge`, which also should have low memory footprint according to the manual pages.
- Renumber the pbf using osmium `renumber`.
- Try to import it using osm2pgsql.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2088#discussioncomment-7223210
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2088/comments/7223210 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231008/b7795f72/attachment.htm>