[OSM-dev] Speeding up Osm2pgsql through parallelization?

[Andy Allan]

On Tue, Sep 13, 2011 at 1:07 AM, Kai Krueger <kakrueger at gmail.com> wrote:
> Hi,
>
> I was thinking about ways to try and speed up osm2pgsql. Currently a good
> fraction of time, both in full imports and during diff-processing, is spent
> in the "going over pending ways / relations" section. Therefore speeding up
> that section should bring the overall time down quite a bit. One thought to
> try and speed up the "going over pending ways / relations" is to try and
> parallelize it.

That's funny, I'd been looking last week at the next step in the
processing and wondering if I could get a speed increase by
un-parrallel-ising it. I don't have the time or the skills to make
much headway, so I'll happily confuse this thread by talking about
other things.

http://trac.openstreetmap.org/browser/applications/utils/export/osm2pgsql/output-pgsql.c?rev=26651#L1365

At the moment, the creation of the temporary tables is done in
parallel, and so you need up to the sum of the sizes of the geometry
tables in free space (albeit some tables run faster than others, so -
depending on timing - you need less space). My concern is that doing
this serially will lead to improved IO (instead of thrashing between
threads) and less free space required since you'll only need up to
max(sizeoftables) instead of potentially sum(sizeoftables).

As for the create tmp -> sort -> overwrite, is there anything to be
gained by using the built-in CLUSTER instead? I'm not sure how well
our method will actually arrange things on-disk, but again I've done
nothing to investigate any hunches.

Just some thoughts from having stared at the output for too many hours :-)

Cheers,
Andy