[OSM-dev] Problems with planet files and bz2

Christian Vetter veaac.fdirct at gmail.com
Tue Mar 1 22:06:19 GMT 2011

On Tue, Mar 1, 2011 at 10:54 PM, Eric Wolf <ebwolf at gmail.com> wrote:
> Has anyone tried reading the bz2 compressed planet file?
> I'm currently trying to figure out if there is something wrong in Python's
> bz2 module, zlib, or the planet files themselves.
> The python BZ2file module lets you read and write bz2 files like normal
> files without decompressing the whole thing. This is key, especially when
> dealing with full-planet which is 500GB+ uncompressed. However, I've found a
> few instances where different people have had issues with the OSM planet
> files and the bz2 module quitting after reading the first 900,000 bytes. Has
> anyone had success with this?
> FYI: The script works perfectly with a 165MB compressed file from Cloudmade.
> -Eric
> -=--=---=----=----=---=--=-=--=---=----=---=--=-=-
> Eric B. Wolf                           720-334-7734
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev

Most likely the module does not handle multi-stream bz2 files on
itself. These files are created by parallel packing programs and if
the module does not handle them correctly it will stop after the end
of the first stream. I would take a look into the documentation
whether it says anything about unpacking multi-stream archives and how
you have to initialize the next stream.


Christian Vetter

More information about the dev mailing list