[OSM-dev] Problems with planet files and bz2

Eric Wolf ebwolf at gmail.com
Tue Mar 1 23:34:49 GMT 2011


Yep. That is the problem. Figured it out from both sides of the equation.

As of Python 2.7.1 and 3.2, the BZ2File object type does not support reading
multi-stream bz2 files, like those generated by pbzip2.

I'm going to have to look into another solution.

-Eric

-=--=---=----=----=---=--=-=--=---=----=---=--=-=-
Eric B. Wolf                           720-334-7734





On Tue, Mar 1, 2011 at 3:06 PM, Christian Vetter <veaac.fdirct at gmail.com>wrote:

> On Tue, Mar 1, 2011 at 10:54 PM, Eric Wolf <ebwolf at gmail.com> wrote:
> > Has anyone tried reading the bz2 compressed planet file?
> > I'm currently trying to figure out if there is something wrong in
> Python's
> > bz2 module, zlib, or the planet files themselves.
> > The python BZ2file module lets you read and write bz2 files like normal
> > files without decompressing the whole thing. This is key, especially when
> > dealing with full-planet which is 500GB+ uncompressed. However, I've
> found a
> > few instances where different people have had issues with the OSM planet
> > files and the bz2 module quitting after reading the first 900,000 bytes.
> Has
> > anyone had success with this?
> > FYI: The script works perfectly with a 165MB compressed file from
> Cloudmade.
> > -Eric
> > -=--=---=----=----=---=--=-=--=---=----=---=--=-=-
> > Eric B. Wolf                           720-334-7734
> >
> >
> >
> >
> > _______________________________________________
> > dev mailing list
> > dev at openstreetmap.org
> > http://lists.openstreetmap.org/listinfo/dev
> >
> >
>
> Most likely the module does not handle multi-stream bz2 files on
> itself. These files are created by parallel packing programs and if
> the module does not handle them correctly it will stop after the end
> of the first stream. I would take a look into the documentation
> whether it says anything about unpacking multi-stream archives and how
> you have to initialize the next stream.
>
> Greetings,
>
> Christian Vetter
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20110301/68519999/attachment.html>


More information about the dev mailing list