[OSM-dev] visible-Flag in PBF

Scott Crosby scott at sacrosby.com
Sat May 7 20:23:33 BST 2011


On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
> Hi,
>
> On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <scott at sacrosby.com> wrote:
>> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
>>> With regard to LZMA: I have some C++ code lying around to compress /
>>> decompress LZMA... I can test how much it would affect file size /
>>> decoding speed.
>>
>> Cool. You don't need a full-fledged PBF reader&writer to test it. Just
>> enough to parse out blobs and write blobs.
>
> I quickly hacked it into MoNav's importer and tested it on the extract
> of Germany. I used maximum compression ( dictionary size == blob size
> ):
>
> size of zlib blobs: 849MB
> size of lzma blobs: 762MB
> time spent decoding zlib blobs: 6.986 s
> time spent decoding lzma blobs: 49.078 s
>
> We can reduce the size a bit by using lzma ( ~10% ) and adding it
> isn't much work ( about 10 lines of code for encoding / decoding ).
> However, it doesn't seem worth it, considering that it makes parsing
> slower. Increasing the block size would most likely not increase the
> compression: I tested compressing all uncompressed blobs at once using
> a 64MB dictionary and the size only decreased to 728MB

Interesting. This doesn't rule out using LZMA. With osmosis,
decompression time is small compared to the other overheads.
(uncompressed files are no faster to read than compressed files), so
the decompression hit isn't too bad. How bad is the compression
performance hit? Deflate halves the compression performance in osmosis
compared to uncompressed.

Ultimately, I think the decision will come down to whether a 10% space
savings is worth adding a dependency on LZMA. Opinions?

Scott



More information about the dev mailing list