[OSM-talk] Something wrong with planet file? Or osmosis?

[Paul Houle]

  On 11/26/2010 11:31 PM, Toby Murray wrote:
>
> I think you are seeing the problem with java's built in bzip2 library. 
> It doesn't support all bzip2 features. Try unzipping the planet file 
> using an external program and piping it into osmosis. Like this 
> (assuming you are on linux):
> bunzip2 planet.bz2 | osmosis --rx /dev/stdin [...]
>
> There is a small note about this on the bottom of the osmosis wiki 
> page but it should probably be a little more prominent somewhere...
>
> Toby
>
     I've had the same problem with a third-party BZip2 library for .NET 
as well.  Of course,  there's something in .NET that works like popen() 
in unix,  so it's not hard at all to use GNU bzip,  which will accept 
pbzip2's output.

     That said,  my new strategy for dealing with "large dump files" is 
to cut the file into segments (like 'split') and recompress the 
fragments.  If your processing chain allows it,  this can be a powerful 
way to get a concurrency speedup.  If more dump files were published in 
this format,  we could get the benefits of "parallel compression" 
without the cost.