[osmosis-dev] Moving to commons-compression from org.apache.tools.bzip2

Alexander Chmyr achmyr at cloudmade.com
Tue Dec 8 09:27:11 GMT 2009


Hello Hakan,

It would be great if you'll perform load testing for bzip2 archives. We're
using osmosis to process planet osm file and had 3 options to use here:
1. internal osmosis bzip2 lib
2. bunzipped file
3. feed osmosis with decompressed stream ( bzip2 -dc | osmosis --rx - --wx -
| bzip )

also there's pbzip option, but it requires pbzipped archive to process.

Our choise was 3. because it has about 30% advantage for 1st option, but,
eventually it will not work for windows env.

So, could you check, if new library takes less time to bunzip file, then we
could advice to others best way to process archives.

Regards,
Alexander Chmyr
Senior Software Engineer
Cogniance, Inc


On Tue, Dec 8, 2009 at 3:10 AM, Brett Henderson <brett at bretth.com> wrote:

> On Tue, Dec 8, 2009 at 12:01 PM, Hakan Tandogan <hakan at gurkensalat.com>wrote:
>
>> Brett Henderson wrote:
>>
>>  On Tue, Dec 8, 2009 at 11:01 AM, Hakan Tandogan <hakan at gurkensalat.com<mailto:
>>> hakan at gurkensalat.com>> wrote:
>>>
>>> If you can then type "ant resolve" and have it run successfully it's
>>> working.
>>>
>>
>> Hmmm... Ivy should have found the archive on repo1.maven.org .
>> mirrors.ibiblio.org has that artifact though (where my maven installation
>> got it from).
>>
>
> Um, repo1.maven.org looks okay but I can't get it to download either ...
>
>
>> Is there a way to convince ivy to fetch it from ibiblio?
>
>
> There is, but you have to configure ivy to use multiple public repos.  It's
> not terribly difficult but requires creating a couple of extra config
> files.  The current public repo config is out of the box.  I should be able
> to take a look at it tonight if you like.
>
>
>>
>>  Yep, a unit test would be helpful.  I guess a simple enough test would be
>>> to just write some data to a temporary compressed file and make sure you can
>>> read it back again.  Perhaps manually uncompress the file the first time
>>> with command line bzip2 just to double check it's doing compression.
>>>
>>
>> I need to find a smallish osm file and use it for unit testing. Let me
>> search on my filesystems a little bit :-)
>
>
> There's a few already in the source tree under test/data/template/v0_6.
> You could copy one of those ...
>
>
>>
>>
>>  The osmosis.bat launch script will also need to be updated to point to
>>> the new lib.
>>>
>>
>> Actually, not anymore :-)
>>
>> I just committed my changes to use the classworlds launcher, it is
>> configured to collect all jars in lib/default
>>
>
> Great!  I'll take a look later.
>
> Brett
>
>
> _______________________________________________
> osmosis-dev mailing list
> osmosis-dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/osmosis-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmosis-dev/attachments/20091208/1587d241/attachment.html>


More information about the osmosis-dev mailing list