<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hello,<div class=""><br class=""></div><div class="">Can anyone provide anecdotes of use cases where multi-threaded PBF reading significantly speeds up processing? Generally I would expect PBF reading to be IO bound rather than processor bound, but I still need to make more accurate measurements. </div><div class=""><br class=""></div><div class="">Of course actually processing the OSM data once the PBF is decoded can be quite CPU intensive, but that would imply buffering decoded data and parallelizing geometric operations for example, not the reading.</div><div class=""><br class=""></div><div class="">I’d appreciate any data points and example use cases you might have, as I’m currently working on related tooling.</div><div class=""><br class=""></div><div class="">Andrew Byrd</div><div class=""><br class=""><div class=""><div><blockquote type="cite" class=""><div class="">On 04 Jun 2015, at 05:57, Brett Henderson <<a href="mailto:brett@bretth.com" class="">brett@bretth.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On 30 April 2015 at 03:27, Paul Norman <span dir="ltr" class=""><<a href="mailto:penorman@mac.com" target="_blank" class="">penorman@mac.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 4/29/2015 9:55 AM, Martijn van Exel wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If osmosis is the reference implementation, is there a reason why it<br class="">
doesn't seem to leverage this block structure to speed up reading? Or<br class="">
does it?<br class="">
</blockquote></span>
Osmosis has the --read-pbf-fast task which allows multiple worker threads.<br class=""></blockquote><div class=""><br class=""></div><div class="">That's right. I forget how the PBF structure works off the top of my head, but the file is already split into blocks. The main --read-pbf-fast thread simply grabs the outer protobuf blocks from file and then distributes them to worker threads who parse out the OSM entities from within the block. After extraction, the entities within each block are passed to the downstream task in original file order. I'm not sure I see the need to modify the PBF file format.</div><div class=""> </div></div></div></div>
_______________________________________________<br class="">osmosis-dev mailing list<br class=""><a href="mailto:osmosis-dev@openstreetmap.org" class="">osmosis-dev@openstreetmap.org</a><br class="">https://lists.openstreetmap.org/listinfo/osmosis-dev<br class=""></div></blockquote></div><br class=""></div></div></body></html>