[OSM-dev] [osmosis-dev] Proposal for a multithreaded PBF reader

[Brett Henderson]

On 30 April 2015 at 03:27, Paul Norman <penorman at mac.com> wrote:

> On 4/29/2015 9:55 AM, Martijn van Exel wrote:
>
>> If osmosis is the reference implementation, is there a reason why it
>> doesn't seem to leverage this block structure to speed up reading? Or
>> does it?
>>
> Osmosis has the --read-pbf-fast task which allows multiple worker threads.
>

That's right.  I forget how the PBF structure works off the top of my head,
but the file is already split into blocks.  The main --read-pbf-fast thread
simply grabs the outer protobuf blocks from file and then distributes them
to worker threads who parse out the OSM entities from within the block.
After extraction, the entities within each block are passed to the
downstream task in original file order.  I'm not sure I see the need to
modify the PBF file format.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20150604/3c9fb3f1/attachment.html>