[OSM-dev] Simpler binary OSM formats
francois.battail at sipibox.fr
Wed Apr 29 09:45:55 UTC 2015
Le 29/04/2015 11:18, Jochen Topf a écrit :
> Looking at your proposal you seem to be very concerned with file size but not
> so much with read/write speed. From my experience reading and writing PBF is
> always CPU bound. Removing complexity could speed this up considerably. But
> if the price is that we need zlib (de)compression it might not be worth it,
> because it is rather CPU and memory intensive. Currently you can save quite
> a lot of CPU time if you do not compress the PBF blocks but leave them
Exactly. From my soon to be released code:
"Tests were made on planet-141112.osm.pbf (28406347563 bytes) with an
AMD FX 8350 (8 cores) 4 GHz, 32 GB, data on a WD Red 2TB SATA 3 disk,
system: Ubuntu 14.04.1 installed on an Intel SSD SATA 3.
No inserts were made on the database or dumps, only pbf parsing.
reference (time cp planet-141112.osm.pbf /dev/null): 205s (bandwidth:
with liprotobuf-c.so, default allocator, without assembly support: 799s
(bandwidth: 33.9 MiB/s)
with liprotobuf-c.so, sw_pool_t allocator, little assembly support: 629s
(bandwidth: 43.1 MiB/s)"
Most of the time is spent in zlib, libprotobuf-c and memory allocations.
I've addressed the last point using x86_64 assembly language and a pool
allocator. I think a rewrite of an optimized libprotobuf library would
help to gain some speed but the cost is very high (at least for my
Better advice I can give: use SSD ;-)
More information about the dev