[OSM-dev] visible-Flag in PBF

[Scott Crosby]

On Sat, May 7, 2011 at 3:01 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
> On Sat, May 7, 2011 at 9:23 PM, Scott Crosby <scott at sacrosby.com> wrote:
>> On Sat, May 7, 2011 at 1:25 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
>>> Hi,
>>>
>>> On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <scott at sacrosby.com> wrote:
>>>> On Fri, May 6, 2011 at 9:24 PM, Christian Vetter <veaac.fdirct at gmail.com> wrote:
>>>>> With regard to LZMA: I have some C++ code lying around to compress /
>>>>> decompress LZMA... I can test how much it would affect file size /
>>>>> decoding speed.
>>>>
>>>> Cool. You don't need a full-fledged PBF reader&writer to test it. Just
>>>> enough to parse out blobs and write blobs.
>>>
>>> I quickly hacked it into MoNav's importer and tested it on the extract
>>> of Germany. I used maximum compression ( dictionary size == blob size
>>> ):
>>>
>>> size of zlib blobs: 849MB
>>> size of lzma blobs: 762MB
>>> time spent decoding zlib blobs: 6.986 s
>>> time spent decoding lzma blobs: 49.078 s
>>>
>>> We can reduce the size a bit by using lzma ( ~10% ) and adding it
>>> isn't much work ( about 10 lines of code for encoding / decoding ).
>>> However, it doesn't seem worth it, considering that it makes parsing
>>> slower. Increasing the block size would most likely not increase the
>>> compression: I tested compressing all uncompressed blobs at once using
>>> a 64MB dictionary and the size only decreased to 728MB
>>
>> Interesting. This doesn't rule out using LZMA. With osmosis,
>> decompression time is small compared to the other overheads.
>> (uncompressed files are no faster to read than compressed files), so
>> the decompression hit isn't too bad. How bad is the compression
>> performance hit? Deflate halves the compression performance in osmosis
>> compared to uncompressed.
>
> About 312s to compress all blobs for Germany. Changing the dictionary
> size does not change much. I lowered it all the way down to 64kb and
> the values stayed the same essentially.
>

And deflate?

Scott