<div>On Thu, Sep 30, 2010 at 6:04 AM, Stefan de Konink <span dir="ltr"><<a href="mailto:stefan@konink.de">stefan@konink.de</a>></span> wrote:</div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Thu, 30 Sep 2010, Scott Crosby wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I proposed exactly this for the mkgmap splitter. If you're going to do this,<br>
can I propose a tweak where you can output thousands of files<br>
simultaneously? The differences are minor: Instead of tracking if a<br>
node/way/relation was output or was missed with two bitsets, track which<br>
areas it has been dumped to with two multimaps from the ID to the list of<br>
areas it was output to or the list of areas in which it was missed.<br>
</blockquote>
<br></div>
How would you a priori know if a node is in a certain area if you haven't observed its locatio or trace it?<div class="im"><br></div></blockquote><div><br></div><div>You don't know a.priori. However, you can track when which nodes/ways/relations have already been output and which are missing. On the second pass, output the missing ones, and also add to the list of.</div>
<div><br></div><div>But in the first pass, you output nodes&ways&relations. You also track which nodes/ways/relations have already been output, and also track *missing* nodes/ways/relations. (So that you know that they need to be output on the second pass.)</div>
<div><br></div><div>DO:</div><div><br></div><div>while (node = stream.getNode()) {</div><div> nodeid = node.getId();</div><div> // Handle missing nodes.</div><div> for (areaid : missingNodesMultiMap.get(nodeid) {</div>
<div> areas[areaid].writeNode(node);</div><div> alreadyOutputMultiMap.addKeyVal(nodeid,areaid);</div><div> }</div><div> missingNodesMap.clearKey(nodeid)</div><div> // Only on the first pass:</div><div> if (isFirstPass) {</div>
<div> for (area : areas) {</div><div> if area.containsNode(node) {</div><div> area.writeNode(node)</div><div> alreadyOutputMultiMap.addKeyVal(nodeid,area.getId());</div><div> missingNodesMultiMap.remove(node.getId(),area.getId());</div>
<div> }</div><div> }</div><div> }</div><div><br></div><div>while (way = stream.getWay()) {</div><div> wayid = way.getId(); </div><div> // Handle relations reporting us as missing on second and later passes.</div>
<div> for (areaid : missingWaysMultiMap(way)) {</div><div> for (nodeid : way.getNodesIds()) {</div><div> if (!alreadyOutputNodes.contains(nodeid,areaid))</div><div> missingNodes.addKeyVal(wayid,areaid);</div>
<div> }</div><div><div> missingWaysMap.clearKey(wayid)</div></div><div><br></div><div> // Only needed on the first pass to find the inital set of ways in the area ; any way that contains a node in the area in question.</div>
<div> if (isFirstPass) {</div><div> for (area : areas) {</div><div> areaid=arae.getid();</div><div> for (nodeid : way.getNodesIds()) {</div><div> if (alreadyOutputedNodes.contains(nodeid,areaid))</div>
<div> toOutput = true;</div><div> }</div><div><div> for (nodeid : way.getNodesIds()) {</div></div><div> if (toOutput && !alreadyOutputNodes.contains(nodeid,areaid))</div><div> missingNodes.addKeyVal(nodeid,nodeid);</div>
<div> }</div><div> }</div><div>}</div><div><br></div><div>// Relations are similar to ways. Track what has been output and what contained relations/ways/nodes are missing and need to be output on the next pass.</div>
<div><br></div><div>WHILE( missingNodes && missingWays && missingRelations are nonempty);</div><div><br></div><div>From my java benchmarks done with the splitter, a whole planet can be split into 6000 disjoint areas at a time on a 8gb machine. </div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
BBOX?<br>
<br>
Count of nodes/ways/relations in that block?<br>
<br>
What else?<br>
</blockquote>
<br></div>
I would find it very interesting if different types of output could be exported individually. </blockquote><div><br></div><div>Definitely doable, but this needs to be thought through. To preserve the semantics of each node/way/relation is in the file exactly once, the exact partitioning strategy needs to be planned very carefully to guarantee that each node is in exactly one partition, guarantee that all of required partitions can be found, so that the output isn't different, while being efficient and skipping as much of the file as possible. Furthermore, it should work when composed with geographic sorting.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">For example being context aware. Some data is landuse, I don't need landuse for routing, so it might be exported in a completely different part of the pbf. So if the format would be descriptive about 'exclusive roads' that might also help the application that uses the data to extract or leave the set.<br>
<br>
I don't think that counts are useful. The mbr is.</blockquote><div><br></div><div>Veeac requested the counts in the prior message.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ok, Is XML's gzipped size or parsing speed a bottleneck for storing or<br>
processing changes? I'd be happy to offer suggestions on the protocol buffer<br>
architecture.<br>
</blockquote>
<br></div>
>From what I observe now the bottleneck seems to be actually protocol buffers, while my output code can become slightly faster.</blockquote><div><br></div><div>This would imply that doing binary changesets isn't a critical necessity.</div>
<div><br></div><div>Scott</div><div> </div></div><br>