[OSM-dev] Binary Format, Performance, pbf2osm

[Frederik Ramm]

Hi,

    updated according to Scott's request (and subject fixed), comparison 
using the OSM Bavaria extract:

File sizes:

.osm     2462778377
.osm.pbf  134579577 (with compression=deflate and lossless)
.osm.pbf  255103238 (uncompressed)
.osm.bz2  223006298 (-2)
.osm.gz   280224099

Decompression (saving to .osm.xml):

from .osm.bz2 with bunzip:                 0m 57s (user: 0m 52s)
from .osm.bz2 with osmosis:                3m 27s (user: 4m 06s)
from .osm.pbf (deflate) with pbf2osm:      0m 55s (user: 0m 41s)
from .osm.pbf (deflate) with osmosis:      1m 11s (user: 1m 23s)
from .osm.pbf (uncompressed) with pbf2osm: ***
from .osm.obf (uncompressed) osmosis:      1m 18s (user: 1m 18s)
from .osm.gz with gunzip:                  0m 26s (user: 0m 14s)
from .osm.gz with osmosis:                 1m 11s (user: 2m 06s)

Notes:

Disk performance: Writing a file the size of the target file (with dd 
if=/dev/zero) onto the target disks takes about 0m 03s).

All "osmosis" runs done with --buffer bufferCapacity=12000, achieving 
some concurrency in the decompress case.

The *** case couldn't be tested: "Error unpacking PrimitiveBlock 
message". The offending file is at 
http://www.remote.org/frederik/tmp/by-none.osm.pbf if someone wants to 
check what's wrong with it.

The timings are to be taken with a grain of salt, they're just one-off 
spot checks; a more thorough analysis should use several different OSM 
files and run the same program multiple times.

Bye
Frederik