[OSM-dev] Osmosis woes, something not right - can someone pls. see why?
brett at bretth.com
Wed Jul 30 07:16:37 BST 2008
On Wed, Jul 30, 2008 at 1:24 PM, Brett Henderson <brett at bretth.com> wrote:
> 2008/7/26 Karl Newman <siliconfiend at gmail.com>
>> Well, I ran it for a while until it failed because of lack of heap space
>> (I had -Xmx512M for the Java command but it wasn't enough for the
>> ListIdTracker). But regardless, it already generated a 18.6 GB uncompressed
>> OSM file (and that's only nodes), so I suspect there's something wrong with
>> the creation of your segment_earth.osm. Make sure it's not failing silently.
>> A few optimizations you could use:
>> 1. Use planet.gz, not bz2 because Osmosis is much faster at gzip (it
>> has native code).
>> 2. Don't enable date parsing.
>> 3. Add the parameter "idTrackerType=BitSet" to the bounding box task.
>> It's more efficient for large bounding boxes.
> Apologies for the slow reply, I've had a lot on.
> I had similar issues with out of memory errors and had to switch to the
> BitSet id tracker. I ended up with a 3.25 GB gzip compressed file for the
> segment file.
> Another optimisation is to stop using the log progress task. I haven't
> tried running with and without logging recently but I suspect it adds a lot
> of overhead. There's probably a much smarter way of doing this but the
> current implementation isn't very efficient.
> I'll run the second step now.
I've completed the New York extraction from the segment file and created a
71.3MB gzip compressed file, that will be approaching 1GB uncompressed. I
ran this second invocation with idTrackerType=IdList due to the much smaller
number of nodes in the result.
It all seems to be working okay at my end. All I can think of is that the
osmosis process failed unexpectedly due to an out of memory error. The
files being created were too small.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the dev