[Tile-serving] [osm2pgsql-dev/osm2pgsql] Some research on middle performance (Issue #2110)

[mboeringa]

> I believe using postgres as node cache was bad choice in the first place, and minor optimizations wont "save us" here. Postgres has way too much overhead for this purpose, in terms of time, storage space and IOPS. Simple high-performance key-value store library, such as LevelDB, would outperform it significantly.
> 
> For example, Imposm is using LevelDB for node cache, and on my machine (cloud VM, so poor IOPS) building node cache takes about 1 hour, and 160 GB of disk space. osm2pgsql slim mode (with 50GB RAM cache) takes several hours, and I honestly don't know how much disk space, because I tried the latest planet yesterday, and it crashed on low disk space, with about 550 GB free at the start.

@rouen-sk 

You are aware of the [--flat-nodes option](https://osm2pgsql.org/doc/manual.html#flat-node-store) that osm2pgsql has supported for years to store nodes outside the database in an efficient and fast way?

On my machine (non-cloud), it takes only 34m to import all **+12B** nodes of Facebook's Daylight distribution of OpenStreetMap using the `--flat-nodes` option, and the resulting file should be some 160GB as well. That still leaves the ways and relations stored in PostgreSQL for middle, but I am not sure if imposm doesn't store them as well in PostgreSQL?

550GB free space for a Planet import is becoming very tight. Planet is just to big nowadays, and honestly, with good 4TB NVMe PCIe 3-5.x drives now as low as maybe $200-300, shortage of disk space or IOPS should be no excuse for a failed import.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2110#issuecomment-1847755699
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2110/1847755699 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231208/1467547f/attachment.htm>