[OSM-dev] Problems with planet files and bz2
veaac.fdirct at gmail.com
Tue Mar 1 22:06:19 GMT 2011
On Tue, Mar 1, 2011 at 10:54 PM, Eric Wolf <ebwolf at gmail.com> wrote:
> Has anyone tried reading the bz2 compressed planet file?
> I'm currently trying to figure out if there is something wrong in Python's
> bz2 module, zlib, or the planet files themselves.
> The python BZ2file module lets you read and write bz2 files like normal
> files without decompressing the whole thing. This is key, especially when
> dealing with full-planet which is 500GB+ uncompressed. However, I've found a
> few instances where different people have had issues with the OSM planet
> files and the bz2 module quitting after reading the first 900,000 bytes. Has
> anyone had success with this?
> FYI: The script works perfectly with a 165MB compressed file from Cloudmade.
> Eric B. Wolf 720-334-7734
> dev mailing list
> dev at openstreetmap.org
Most likely the module does not handle multi-stream bz2 files on
itself. These files are created by parallel packing programs and if
the module does not handle them correctly it will stop after the end
of the first stream. I would take a look into the documentation
whether it says anything about unpacking multi-stream archives and how
you have to initialize the next stream.
More information about the dev