[Tile-serving] [openstreetmap/osm2pgsql] Parallel index creation, to minimize required time (and I/O) (Discussion #1562)

[mboeringa]

I think the benefits in this case will be limited. From my experience using parallel indexing, until we finally have a much faster GIST indexing in PostgreSQL and PostGIS implemented, the ultimate thing determining total indexing time is largely down to indexing the spatial 'way' column.

E.g. when I start 4 consecutive threads (each one potentially using PostgreSQL parallel indexing for btree indexes), and index 50+ columns including a single 'way' column, then the total indexing time is fully determined by the way indexing only (which I deliberately initiate on the first started thread). All other indexes together cost less time using the remaining 3 threads.

Additionally, this scenario would only benefit in case you create single table using osm2pgsql. As far as I can tell, osm2pgsql will already index multiple tables in parallel, so the benefits of parallel indexing on a single table will be less (and may hit limits of parallel workers in PostgreSQL).

So this would likely only really make sense in the case of a style defining a single table.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/discussions/1562#discussioncomment-1363993
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210921/17f00af8/attachment.htm>