[Tile-serving] [osm2pgsql-dev/osm2pgsql] Any reasons why osm2pgsql doesn't leverage parallelization during huge maps import (eg: whole world)? (Discussion #2090)

[Paul Norman]

I don't see an issue here. I did a `--slim --drop` import on a 2017 consumer machine with NVMe (i7-8700, 64G ram, 2x 1TB NVMe SSDs, untuned PostgreSQL) osm2pgsql took 10h32m to import the planet. 9h of this was processing, with most of the remainder being clustering and index building. It'd be nice to get these times down, but half a day on a 5 year old machine is reasonable for the planet. With a ten year old system and low-end CPU I'd expect yours to take longer, of course.

You haven't provided enough information to show what the bottleneck is, but it's almost certainly the disks, regardless of what you've said. Disk measurements in MB/s are not useful and indicate a misunderstanding of random IO workloads.

A system running osm2pgsql will benefit from being able to use lots of RAM. This RAM is not directly used by osm2pgsql, but is used by the OS to cache.

You also haven't indicated how you adjusted your PostgreSQL settings. This can make a significant difference in performance in the postprocessing stage.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2090#discussioncomment-7193411
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2090/comments/7193411 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231004/a60856d7/attachment.htm>