[OSM-dev] visible-Flag in PBF

Christian Vetter veaac.fdirct at gmail.com
Sat May 7 21:01:26 BST 2011


On Sat, May 7, 2011 at 9:23 PM, Scott Crosby <scott at sacrosby.com> wrote:
> On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
>> Hi,
>>
>> On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <scott at sacrosby.com> wrote:
>>> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
>>>> With regard to LZMA: I have some C++ code lying around to compress /
>>>> decompress LZMA... I can test how much it would affect file size /
>>>> decoding speed.
>>>
>>> Cool. You don't need a full-fledged PBF reader&writer to test it. Just
>>> enough to parse out blobs and write blobs.
>>
>> I quickly hacked it into MoNav's importer and tested it on the extract
>> of Germany. I used maximum compression ( dictionary size == blob size
>> ):
>>
>> size of zlib blobs: 849MB
>> size of lzma blobs: 762MB
>> time spent decoding zlib blobs: 6.986 s
>> time spent decoding lzma blobs: 49.078 s
>>
>> We can reduce the size a bit by using lzma ( ~10% ) and adding it
>> isn't much work ( about 10 lines of code for encoding / decoding ).
>> However, it doesn't seem worth it, considering that it makes parsing
>> slower. Increasing the block size would most likely not increase the
>> compression: I tested compressing all uncompressed blobs at once using
>> a 64MB dictionary and the size only decreased to 728MB
>
> Interesting. This doesn't rule out using LZMA. With osmosis,
> decompression time is small compared to the other overheads.
> (uncompressed files are no faster to read than compressed files), so
> the decompression hit isn't too bad. How bad is the compression
> performance hit? Deflate halves the compression performance in osmosis
> compared to uncompressed.

About 312s to compress all blobs for Germany. Changing the dictionary
size does not change much. I lowered it all the way down to 64kb and
the values stayed the same essentially.

Regards,

Christian Vetter



More information about the dev mailing list