[Tile-serving] [osm2pgsql-dev/osm2pgsql] Any reasons why osm2pgsql doesn't leverage parallelization during huge maps import (eg: whole world)? (Discussion #2090)

[mboeringa]

> Any particular reason(s) why the resources are not utilized at their max pottential ?
> 
> Thanks!
> 
> PS: the system I'm running on is a 22 cores 120GB RAM VM running on a server with 2 x Xeon v2 (24 cores total) with 128GB of ram and 2 x 1TB SATA SSD in RAID 1.

Apart from possible other reasons within osm2pgsql, I strongly recommend you to go with at least a Xeon E5 v4 processor that supports PCIe 3.0 if you want to be on a budget second hand server instead of v2, _and get off SATA for your disks_. Disk IO is probably one of the main bottlenecks in such import processes.

I import the entire Planet using a complex flex style in [about 8 hours on my 2016 age HP Z840](https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks#Planet_(75.1GB_PBF,_2023-07-17)_on_HP_Z840_workstation,_512GB_DDR4_2133,_2x_Xeon_E5-2699v4_22C,_5x_Samsung_970_EVO_Plus_2TB_NVMe_RAID_0,_4x_Intel_OPTANE_P1600X_110GB_NVMe_RAID_0,_Ubuntu_22.04_(flex_|_--slim)_~8h), using regular PCIe NVMe disks and a cheap PCIe 3.0 NVMe card with bifurcation set in the system's BIOS. Runs highly reliable.

Sure, the switch to NVMe will not suddenly make your system go 100% on CPU and IO all the time, but it does help with IO bottlenecks.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2090#discussioncomment-7187672
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2090/comments/7187672 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231004/bf5738e3/attachment.htm>