[OSM-dev] Osmosis woes, something not right - can someone pls. see why?

[Brett Henderson]

On Wed, Jul 30, 2008 at 1:24 PM, Brett Henderson <brett at bretth.com> wrote:

> 2008/7/26 Karl Newman <siliconfiend at gmail.com>
>
>>
>> Well, I ran it for a while until it failed because of lack of heap space
>> (I had -Xmx512M for the Java command but it wasn't enough for the
>> ListIdTracker). But regardless, it already generated a 18.6 GB uncompressed
>> OSM file (and that's only nodes), so I suspect there's something wrong with
>> the creation of your segment_earth.osm. Make sure it's not failing silently.
>>
>> A few optimizations you could use:
>>
>>    1. Use planet.gz, not bz2 because Osmosis is much faster at gzip (it
>>    has native code).
>>    2. Don't enable date parsing.
>>    3. Add the parameter "idTrackerType=BitSet" to the bounding box task.
>>    It's more efficient for large bounding boxes.
>>
>> Karl
>>
>
> Apologies for the slow reply, I've had a lot on.
>
> I had similar issues with out of memory errors and had to switch to the
> BitSet id tracker.  I ended up with a 3.25 GB gzip compressed file for the
> segment file.
>
> Another optimisation is to stop using the log progress task.  I haven't
> tried running with and without logging recently but I suspect it adds a lot
> of overhead.  There's probably a much smarter way of doing this but the
> current implementation isn't very efficient.
>
> I'll run the second step now.
>
>
I've completed the New York extraction from the segment file and created a
71.3MB gzip compressed file, that will be approaching 1GB uncompressed.  I
ran this second invocation with idTrackerType=IdList due to the much smaller
number of nodes in the result.

It all seems to be working okay at my end.  All I can think of is that the
osmosis process failed unexpectedly due to an out of memory error.  The
files being created were too small.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20080730/e8f3b753/attachment.html>