[OSM-dev] osmosis bz2 performance
stefan.baebler at gmail.com
Wed Feb 6 14:23:34 GMT 2008
On Feb 6, 2008 2:44 PM, Brett Henderson <brett at bretth.com> wrote:
> Hi Stefan,
> The apache bz2 implementation was the only version I could find, if
> there's an alternative I'd like to hear about it. The apache
> implementation is pure java and I suspect that is the main difference.
> The gzip implementation uses Inflater and Deflator classes which have a
> bunch of native methods. I see the bz2 support in osmosis as being a
I was suspecting this, tnx for confirmation.
> convenience aid, but if I'm processing large files I use the native
> bzip2 command line tools and pipe the data into osmosis. For example.
Probably a brief note in the Wiki could save quite a bit of someone's
time. Added to wiki.
> bzcat planet.bz2 | osmosis --rx /dev/stdin --wn
Nice on unixes, but I can't seem to read "/dev/stdin" in Cygwin.
I tried that to pipe the planet straight from wget -> osmosis without
writing the planet to file.
putting native bzip in between would improve perofrmance quite a bit.
wget -O - http://.... | bzip -c | osmosis -rx /dev/stdin ...
Can Osmosis get some alias for /dev/stdin (eg "-" would mean stdin on
reading tasks and stdout on writing tasks, or we can reserve and
handle differently "stdin" and "stdout" file names) ?
If we agree on this design i can implement it.
More information about the dev