[OSM-dev] Simpler binary OSM formats

[François Battail]

Le 29/04/2015 11:18, Jochen Topf a écrit :

> Looking at your proposal you seem to be very concerned with file size but not
> so much with read/write speed. From my experience reading and writing PBF is
> always CPU bound. Removing complexity could speed this up considerably. But
> if the price is that we need zlib (de)compression it might not be worth it,
> because it is rather CPU and memory intensive. Currently you can save quite
> a lot of CPU time if you do not compress the PBF blocks but leave them
> uncompressed.[...]

Exactly. From my soon to be released code:

"Tests were made on planet-141112.osm.pbf (28406347563 bytes) with an
AMD FX 8350 (8 cores) 4 GHz, 32 GB, data on a WD Red 2TB SATA 3 disk,
system: Ubuntu 14.04.1 installed on an Intel SSD SATA 3.
No inserts were made on the database or dumps, only pbf parsing.

reference (time cp planet-141112.osm.pbf /dev/null): 205s (bandwidth: 
132 MiB/s)

with liprotobuf-c.so, default allocator, without assembly support: 799s 
(bandwidth: 33.9 MiB/s)
with liprotobuf-c.so, sw_pool_t allocator, little assembly support: 629s 
(bandwidth: 43.1 MiB/s)"

Most of the time is spent in zlib, libprotobuf-c and memory allocations. 
I've addressed the last point using x86_64 assembly language and a pool 
allocator. I think a rewrite of an optimized libprotobuf library would 
help to gain some speed but the cost is very high (at least for my 
application).

Better advice I can give: use SSD ;-)

Best regards