[OSM-dev] Binary Format, Performance, pbf2osm
frederik at remote.org
Sun Oct 17 10:16:11 BST 2010
updated according to Scott's request (and subject fixed), comparison
using the OSM Bavaria extract:
.osm.pbf 134579577 (with compression=deflate and lossless)
.osm.pbf 255103238 (uncompressed)
.osm.bz2 223006298 (-2)
Decompression (saving to .osm.xml):
from .osm.bz2 with bunzip: 0m 57s (user: 0m 52s)
from .osm.bz2 with osmosis: 3m 27s (user: 4m 06s)
from .osm.pbf (deflate) with pbf2osm: 0m 55s (user: 0m 41s)
from .osm.pbf (deflate) with osmosis: 1m 11s (user: 1m 23s)
from .osm.pbf (uncompressed) with pbf2osm: ***
from .osm.obf (uncompressed) osmosis: 1m 18s (user: 1m 18s)
from .osm.gz with gunzip: 0m 26s (user: 0m 14s)
from .osm.gz with osmosis: 1m 11s (user: 2m 06s)
Disk performance: Writing a file the size of the target file (with dd
if=/dev/zero) onto the target disks takes about 0m 03s).
All "osmosis" runs done with --buffer bufferCapacity=12000, achieving
some concurrency in the decompress case.
The *** case couldn't be tested: "Error unpacking PrimitiveBlock
message". The offending file is at
http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to
check what's wrong with it.
The timings are to be taken with a grain of salt, they're just one-off
spot checks; a more thorough analysis should use several different OSM
files and run the same program multiple times.
More information about the dev