[Tile-serving] [osm2pgsql-dev/osm2pgsql] Any reasons why osm2pgsql doesn't leverage parallelization during huge maps import (eg: whole world)? (Discussion #2090)

[mboeringa]

> SSD max limit is about 600 MB/s

The random IO needed by osm2pgsql is not so much about maximum throughput, but more about latency and IOPS, where NVMe wins over SATA, e.g see below link for an explanation:

https://www.techtarget.com/searchstorage/feature/NVMe-SSD-speeds-explained

I also don't see anywhere near maximum throughput in GBs of my NVMe drives when osm2pgsql runs, but it still is a lot faster due to high random read/write. This is more general an issue with databases, they often need good low queue depth random read/write, which even with NVMe is **_much_** lower than the GBs maximum serial read/write.

E.g. even my raided NVMe Samsung drives have only about 42 MB/s random read at queue depth 1 and with 1 thread (Q1T1), AND 78 MB/s random write.

This ameliorates much with higher queue depth and more threads: e.g. 1774 MB/s random read and 1390 MB/s random write at Q32T16.

That is however still far below the maximum 11GB/s sequential read/write of the same drives as measured in CrystalDiskMark at Q8T1.


-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/discussions/2090#discussioncomment-7191365
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/repo-discussions/2090/comments/7191365 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20231004/1c9436ec/attachment-0001.htm>