[OSM-dev] [osmosis-dev] Proposal for a multithreaded PBF reader

[François Battail]

Le 04/06/2015 11:04, Andrew Byrd a écrit :

Hello Andrew,

> Can anyone provide anecdotes of use cases where multi-threaded PBF
> reading significantly speeds up processing? Generally I would expect PBF
> reading to be IO bound rather than processor bound, but I still need to
> make more accurate measurements.

Here are simple figures:

cp planet.osm.pbf /dev/null: 205s
planet.osm.pbf parsing(*): 629s

(*)With some optimization (memory allocations), without: 799s

If I/O capacity is not an issue, using two threads may improve things 
but from my experience if you're doing something other than a simple 
dump with filtering like putting OSM data into a database there will be 
none to little gain (and you will take useful resources).

> Of course actually processing the OSM data once the PBF is decoded can
> be quite CPU intensive, but that would imply buffering decoded data and
> parallelizing geometric operations for example, not the reading.
Some processing may require to wait for a specific object so the locking 
between reader threads could be rather complex. At least if it's only 
for buffering it's OK but then some memory will be needed...

Best regards