[OSM-dev] Compression types in PBF Format

[Scott Crosby]

On Tue, Nov 30, 2010 at 2:21 AM, Jochen Topf <jochen at remote.org> wrote:
> The PBF format supports three compression types: zlib, lzma, and bzip2. Do
> we have to support all of them? What is the currently existing software
> using?

Good question. I think that the bzip2 compression option is useless.
Too slow, especially on the decompression side. I'm not sure what to
do about LZMA. It offers higher compression ratio's at little loss in
decompression speed. The catch is that while everything supports
deflate, LZMA decompressor support is a lot less widespread. It might
be a valuable future option, but it is also untested.

To my knowledge, Osmosis has the only implementation of a PBF writer,
it only uses uncompressed and zlib? Has anyone else implemented a
writer?

If nobody else has their own writer, then would anyone object to me
unilaterally removing (not depreciating) bzip2 entirely, and
disabling/marking lzma as a proposed future extension?   If tests in
the future show that LZMA offers significant size decreases, it can be
enabled and support can be added.

> IMHO it would make more sense to just define one and stick with it. Easier
> to implement for everybody, less reliance on external libs.

Agreed. Insofar that a reader has to support every permutation of the
format, reducing those permutations is important. I think LZMA has
something to offer, but it needs to be tested first.

Scott