<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>Thanks for your comments Jochen. Responses in-line below.<br></div>
<div> </div>
<div>On Wed, Apr 29, 2015, at 11:18, Jochen Topf wrote:<br></div>
<blockquote type="cite"><div>On Wed, Apr 29, 2015 at 01:35:29AM +0200, andrew byrd wrote:<br></div>
<div>... it has some nice properties we shouldn't forget. First and foremost that is the block structure. This allows generating and parsing in multiple threads. I think thats an important optimization going forward.<br></div>
</blockquote><div> <br></div>
<div>A very good point. In any further testing I will apply intra-block compression rather than compressing the entire stream.<br></div>
<div> </div>
<blockquote type="cite"><div> I also think it is important to have some kind of header for storing file meta data in a flexibly way, PBF has that.<br></div>
</blockquote><div> </div>
<div>Agreed. Any header metadata would be defined later if there is wider interest in the format, since it represents a very small constant factor in performance.<br></div>
<div> </div>
<blockquote type="cite"><div>Looking at your proposal you seem to be very concerned with file size but<br></div>
<div>not so much with read/write speed. From my experience reading and writing PBF is always CPU bound. Removing complexity could speed this up considerably.<br></div>
</blockquote><div> </div>
<div>My true goal was to reduce complexity while at least maintaining the performance characteristics of PBF. Given that goal, it is true that there is too much emphasis on file size in the article. I will need to do a follow-up to cover speed and memory usage.<br></div>
<div> </div>
<div>Anecdotally, I was seeing about 2x speedup relative to PBF writing for a specific processing step, with both PBF and VEX writing code written in C. That certainly needs to be confirmed more methodically.<br></div>
<div> </div>
<div>The PBF parsing code produced by the Protobuf compiler seems to involve quite a lot of dynamic memory allocations. It is straightforward to read and write OSM data with no dynamic allocation at all (outside the compression library), and this is one place where VEX could offer an advantage. <br></div>
<div> </div>
<blockquote type="cite"><div>Currently you can save quite a lot of CPU time if you do not compress the PBF blocks but leave them uncompressed. Of course the file size goes up, but if you have the storage space that doesn't matter that much.<br></div>
</blockquote><div> </div>
<div>I have to admit I did not really consider this because I've rarely encountered uncompressed PBF "in the wild" and have always used PBF with compression turned on. Indeed, any future comparison should ideally include PBF with uncompressed blocks.<br></div>
<div> </div>
<div>With storage space and bandwidth as high as it is today, it is true that throughput should be considered as much or more so than file size. Again, my true motivation is simplicity, and in performance improvements to the extent that they can be enabled by simplicity.<br></div>
<div> </div>
<div>I have observed that speed numbers can of course be quite different depending on whether you handle the compression in a separate thread.<br></div>
<div> </div>
<blockquote type="cite"><div>First, I'd like to see the numbers for the whole planet. A size difference between small extracts doesn't really matter all that much, because the absolute size is so small. Savings on the whole planet file would be much more interesting.<br></div>
</blockquote><div> </div>
<div>A good suggestion. After I've amended the VEX format taking into account the commentary I've received, I will perform a test on the whole planet.<br></div>
<div> </div>
<blockquote type="cite"><div>Second: The XML and PBF format usually contain the metadata that you<br></div>
<div>removed in your VEX format. Have you accounted for that in your numbers? Ie. did you remove the metadata from XML and PBF, too?<br></div>
</blockquote><div> </div>
<div>Yes, I stripped the metadata from the source PBF before converting it to all the other formats to ensure a fair comparison. These numbers are of more interest to me in my daily applications, but since I see that there is some interest from the wider community I agree that a future comparison should be done including metadata. <br></div>
<div> </div>
<blockquote type="cite"><div>Incidentally I came up with a similar text format as you did. It is<br></div>
<div>documented here:<br></div>
<div><a href="http://osmcode.org/libosmium/manual/libosmium-manual.html#opl-object-per-line-format">http://osmcode.org/libosmium/manual/libosmium-manual.html#opl-object-per-line-format</a><br></div>
</blockquote><div> </div>
<div>Yes, it's a very similar idea but of course mine is less complete since it began life as debug output. One unusual ingredient of my text output is the inclusion of nodes' complete data inline after the ways that reference them. It's a trade-off which denormalizes intersection nodes, but avoids keeping nodes far away from their references, avoids repeatedly mentioning non-intersection node identifiers, and makes the output human-readable as a series of complete way descriptions.<br></div>
<div> </div>
<div>-Andrew<br></div>
</body>
</html>