[OSM-dev] Memory error while converting osm to gml

[Martijn van Oosterhout]

2009/4/4 Matt Amos <zerebubuth at gmail.com>:
> 2009/4/4 Martijn van Oosterhout <kleptog at gmail.com>:
>> Just reading this again made me realise that we could probably make
>> osm2pgsql faster for the common case just by forking and using
>> gzip/bzip2 to do the decompression in a separate process (i.e. another
>> CPU)... Not entirely sure how much you'd save though (ISTR someone
>> saying it was the polygon construction that took a lot of time), but
>> it'd surely help somewhat.
>
> i found that using unix pipes (i.e: bzcat | osm2pgsql -) for this
> didn't take any less time, probably because bzcat/osm2pgsql are both
> using synchronous reads and writes. maybe if decompression was on a
> different thread using a pair of synchronised buffers?

Using unix pipes is not going to be any slower than using threads, so
if it doesn't help then it doesn't help. Named pipe makes no
difference, it's exactly the same code. Kinda confirms my suspicion
that the decompression is not the bottleneck and that the whole
process is actually blocking elsewhere. My guess: either the DB or the
geometry creation.

Have a nice day,
-- 
Martijn van Oosterhout <kleptog at gmail.com> http://svana.org/kleptog/