[OSM-dev] Simpler binary OSM formats

[andrew byrd]

On Wed, Apr 29, 2015, at 11:45, François Battail wrote:
> with liprotobuf-c.so, default allocator, without assembly support:
> 799s (bandwidth: 33.9 MiB/s) with liprotobuf-c.so, sw_pool_t
> allocator, little assembly support: 629s (bandwidth: 43.1 MiB/s)"

Point taken -- in future comparisons I will focus on throughput as much
or more than size itself. I did notice that protobuf-c allows for
customizing ProtobufCAllocator, and it's interesting to see the results
of using a pool allocator. However my first reaction is that we really
shouldn't need to do much dynamic allocation at all to handle OSM data.
The use of dynamic allocation is a result of the general-purpose nature
of Protobuf. My sense is that Protobuf is not even a particularly good
fit for bulk OSM data transfer, considering the hoops that we must jump
through in dense-nodes to bypass the natural mapping of one Protobuf
message to one OSM entity.

> Most of the time is spent in zlib, libprotobuf-c and memory
> allocations. I've addressed the last point using x86_64 assembly
> language and a pool allocator. I think a rewrite of an optimized
> libprotobuf library would help to gain some speed but the cost is very
> high (at least for my application).

The alternative being to use a format that is not based on Protobuf at
all, but only uses its variable-byte encoding scheme.

Thanks for your comments! Andrew

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20150429/8eee9d1c/attachment.html>