[Tile-serving] [osm2pgsql-dev/osm2pgsql] Any reasons why osm2pgsql doesn't leverage parallelization during huge maps import (eg: whole world)? (Discussion #2090)

[ghevge]

Hi,

I'm new to osm2pgsql realm, but still it is impossible not to notice how slow the tool is when trying to import the whole worls map. 

Monitoring the hardware resources, the only resource that seems to be utilized at maximum is the RAM. The rest: CPU and Disks are berely used.

For example when the command below is started, I'm reading in the logs processing rates of:
`Processing: Node(8646182k 1100.0k/s) Way(25045k 11.64k/s) Relation(0 0.0/s)`

command executed:
`sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-                 script /data/style/openstreetmap-carto.lua --number-processes 22 -S /data/style/                 openstreetmap-carto.style /data/region.osm.pbf -C 8192 --flat-nodes /data/databa                 se/flat_nodes.bin`

At these rates, there is no wonder it takes days to set up an openstreetmap system, if you don't own a last gen server.

My CPU load doesn't go beyond 7% and Disks IOs don't exceed 15 MB/s. Only the RAM is maxed out.

To me this looks like some improper use (or better say: no use of  ) parallelization .

Any particular reason(s) why the resources are not utilized at their max pottential ?

Thanks!

PS: the system I'm running on is a 22 cores 120GB RAM VM running on a server with  2 x Xeon v2 (24 cores total) with 128GB of ram and 2 x 1TB  SATA SSD in RAID 1.
There is only this particular VM running on this system



-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2090
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2090 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231003/36532736/attachment-0001.htm>