[OSM-dev] osm2pgsql slow slim import

["Petr Morávek [Xificurk]"]

"Petr Morávek [Xificurk]" napsal(a):
> Hello,
> I've just upgraded osm2pgsql to latest SVN version (previously I was
> running a version from August, not sure which revision exactly) and I'm
> experiencing extreme slow-down in the phase "Going over pending ways".
> The processing is going at rate ~ 0.1k/s.
> With the old version the import of a relatively small extract took
> approximately 1.5 hour, now I'm at 3 hours and the import is still far
> from completed.
> 
> I have postgresql 9.1 and my import command looks like this:
> 
> osm2pgsql --slim -d osm -U osm -p osm --hstore -C 1500 -S ./import.style
> -r pbf ./data.osm.pbf
> 
> As far as I can tell, the bottle-neck is IO due to UPDATE command. I
> have also tried running the import with options '--cache-strategy
> sparse' and/or '--disable-parallel-indexing', but as far as I can tell,
> it has no effect on this phase.
> 
> Any ideas where the problem could be or how to debug this?
> 
> Best regards,
> Petr Morávek aka Xificurk

An update:
I've narrowed it down - the problem appeared in rev26893 (Parralellize
pending ways / relations).
I've tested multiple revisions with a really small extract: 13MB pbf
(1450k nodes, 178k ways, 2250 rels), I used the command stated above.

rev26892: import time ~1min
rev26893: import time ~22min (the phase "going over pending ways" went
at rate 0.11k/s)

I've got installed:
postgresql-9.1.1
postgis-1.5.3
geos-3.2.2

The weird thing is that such a small extract should fit into RAM, so I
have no idea why is the processing so slow.

Petr Morávek aka Xificurk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xificurk.vcf
Type: text/x-vcard
Size: 211 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20111130/0e9f9368/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20111130/0e9f9368/attachment.pgp>