[OSM-dev] osmosis bz2 performance

Stefan Baebler stefan.baebler at gmail.com
Wed Feb 6 14:23:34 GMT 2008


On Feb 6, 2008 2:44 PM, Brett Henderson <brett at bretth.com> wrote:
> Hi Stefan,
>
> The apache bz2 implementation was the only version I could find, if
> there's an alternative I'd like to hear about it.  The apache
> implementation is pure java and I suspect that is the main difference.
> The gzip implementation uses Inflater and Deflator classes which have a
> bunch of native methods.  I see the bz2 support in osmosis as being a
I was suspecting this, tnx for confirmation.

> convenience aid, but if I'm processing large files I use the native
> bzip2 command line tools and pipe the data into osmosis.  For example.
Probably a brief note in the Wiki could save quite a bit of someone's
time. Added to wiki.

> bzcat planet.bz2 | osmosis --rx /dev/stdin --wn
Nice on unixes, but I can't seem to read "/dev/stdin" in Cygwin.
I tried that to pipe the planet straight from wget -> osmosis without
writing the planet to file.
putting native bzip in between would improve perofrmance quite a bit.
wget -O - http://.... | bzip -c | osmosis -rx /dev/stdin ...

Can Osmosis get some alias for /dev/stdin (eg "-" would mean stdin on
reading tasks and stdout on writing tasks, or we can reserve and
handle differently "stdin" and "stdout" file names) ?
If we agree on this design i can implement it.

Stefan




More information about the dev mailing list