[OSM-dev] Speeding up Osm2pgsql through parallelization?
kakrueger at gmail.com
Wed Sep 14 00:22:42 BST 2011
On 9/13/11 10:49 AM, Andy Allan wrote:
> On Tue, Sep 13, 2011 at 1:07 AM, Kai Krueger<kakrueger at gmail.com> wrote:
>> I was thinking about ways to try and speed up osm2pgsql. Currently a good
>> fraction of time, both in full imports and during diff-processing, is spent
>> in the "going over pending ways / relations" section. Therefore speeding up
>> that section should bring the overall time down quite a bit. One thought to
>> try and speed up the "going over pending ways / relations" is to try and
>> parallelize it.
> That's funny, I'd been looking last week at the next step in the
> processing and wondering if I could get a speed increase by
> un-parrallel-ising it.
Well, I guess there are reasons to parallelize some stuff and
de-parallelize other stuff. But overall, I think as long as you don't
end up over using memory, or moving from a sequential read pattern to a
random one, parallelizing things is probably beneficial.
In the stages you are talking about, working memory to do sorts and
similar things might well however be an issue and it might indeed make
sense to do it in sequence.
> I don't have the time or the skills to make
> much headway, so I'll happily confuse this thread by talking about
> other things.
I think skill wise it would be fairly trivial to try it out. Osm2pgsql
already has a fall back to do these stages in sequence for the case that
pthreads aren't supported. As I think currently this is the only place
that threads are used, you should be able to simply undefine pthreads
and recompile osm2pgsql.
> At the moment, the creation of the temporary tables is done in
> parallel, and so you need up to the sum of the sizes of the geometry
> tables in free space (albeit some tables run faster than others, so -
> depending on timing - you need less space). My concern is that doing
> this serially will lead to improved IO (instead of thrashing between
> threads) and less free space required since you'll only need up to
> max(sizeoftables) instead of potentially sum(sizeoftables).
> As for the create tmp -> sort -> overwrite, is there anything to be
> gained by using the built-in CLUSTER instead? I'm not sure how well
> our method will actually arrange things on-disk, but again I've done
> nothing to investigate any hunches.
It probably would be easier to use CLUSTER and then let postgresql sort
out the rest. It would also make it easier to occasionally "re-cluster".
I guess it wouldn't be too difficult to test it out, if only it didn't
take so long to run those tests...
Was there a reason not to use it in the first place?
> Just some thoughts from having stared at the output for too many hours :-)
More information about the dev