[OSM-dev] Osmosis woes, something not right - can someone pls. see why?

[Karl Newman]

On Fri, Jul 25, 2008 at 9:35 AM, Karl Newman <siliconfiend at gmail.com> wrote:

> 2008/7/25 Fire Girl <firegirl at amorous.com>:
>
>>
>> Hey, that is great -- and Cheers!
>>
>> When you get back, it would be interesting to find out, what kinda of
>> results you get from a two stage extraction, using the same commands,
>>
>> java.exe -jar osmosis.jar" --rx "planet-latest.osm.bz2"
>> enableDateParsing="YES" --lp interval="600" --bb left="-108" right="-36"
>> top="90" bottom="0" --wx "segment_earth.osm"
>>
>> and then,
>>
>> java.exe -jar osmosis.jar" --rx "segment_earth.osm"
>> enableDateParsing="YES" --lp interval="600" --bb left="-75" right="-73"
>> top="42" bottom="40" --wx "new_york_area.osm"
>> ^^^ this is what is yielding a 27 MB file from the 2.5 GB
>> segment_earth.osm slice, when according to my test doing this above same
>> command & parameters against a planet.osm yielded almost a 1.3 GB file :|
>>
>> I mean, it is pretty clear I got a very slimmed down segment_earth.osm
>> extraction.... left="-108" right="-36" top="90" bottom="0", is a pretty big
>> cross-section which yielded 2.5 GB.  Which is totally confusing since I got
>> 1.3 GB from left="-75" right="-73" top="42" bottom="40", ... a tiny 2 degree
>> cross segment!  This is driving me nuts! haha
>>
>> BTW, can you perform this test on a Windows Machine like I am? :))
>>
>>
> I'll give it a shot, too on my Windows laptop (I have a copy of planet)
> once I get to work. One thing I would recommend is that you get a new
> version of Osmosis and then stop using the very slow date parsing. The new
> version just preserves the dates as strings.
>
> Karl
>

Well, I ran it for a while until it failed because of lack of heap space (I
had -Xmx512M for the Java command but it wasn't enough for the
ListIdTracker). But regardless, it already generated a 18.6 GB uncompressed
OSM file (and that's only nodes), so I suspect there's something wrong with
the creation of your segment_earth.osm. Make sure it's not failing silently.

A few optimizations you could use:

   1. Use planet.gz, not bz2 because Osmosis is much faster at gzip (it has
   native code).
   2. Don't enable date parsing.
   3. Add the parameter "idTrackerType=BitSet" to the bounding box task.
   It's more efficient for large bounding boxes.

Karl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20080725/90f90aee/attachment.html>