[Tile-serving] [osm2pgsql-dev/osm2pgsql] Daylight Ingest and Slow Deletions (Discussion #2088)

[mboeringa]

@pyoung-sc 

I am slightly baffled by your results on RDS, they look incongruous. 

On the one hand, you have good node and way loading speeds at 6M/s and 76k/s, but than relation loading is poor at just 117/s. That might be a consequence of some really complex relation processing though, and indeed if I look at the linked Tilezen flex style you referenced, it appears to do some pretty significant processing on relations. So that might explain it.

However, what I don't understand is the really slow "postprocessing stages", they shouldn't take dozens of hours.

E.g. my last Daylight v1.32 import, admittedly with a less complex style but all the other data, only took 14 hours total... See [this benchmark](https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks#Facebook_Daylight_v1.32_(149.3GB_PBF,_2023-10-11)_on_HP_Z840_workstation,_512GB_DDR4_2133,_2x_Xeon_E5-2699v4_22C,_5x_Samsung_970_EVO_Plus_2TB_NVMe_RAID_0,_4x_Intel_OPTANE_P1600X_110GB_NVMe_RAID_0,_Ubuntu_22.04_(flex_|_--slim)_~14h).

Admittedly, this is on a dedicated system, but being a "2016" HP Z840 workstation, it is far from latest hardware (although I did significantly upgrade it with NVMe). Still, I would expect a system capable of achieving 6M/s node and 76k/s way loading, as in your case and very close to the results of node and way loading speeds on my workstation, to have better postprocessing times.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2088#discussioncomment-7368441
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2088/comments/7368441 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231024/0ddf10e2/attachment.htm>