<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Dmitry,<div class=""><br class=""></div><div class="">Yes, there are similarities and I did study the o5m format before I began work on vex. The last section of my original article compares the two and gives my impressions of o5m: <a href="http://conveyal.com/blog/2015/04/27/osm-formats#comparisons-with-o5m" class="">http://conveyal.com/blog/2015/04/27/osm-formats#comparisons-with-o5m</a></div><div class=""><br class=""></div><div class="">In summary: o5m uses string tables with a fixed size and an LRU eviction policy. Producers and consumers must keep their string tables exactly in sync. Strings are then referenced by integers indicating how recently they were used (1 to 15000). This adds quite a bit of complexity to o5m implementations, especially considering that this eviction strategy can backfire on certain inputs leading to files that are actually bigger than a basic gzipped text representation of the same data. According to <a href="http://wiki.openstreetmap.org/wiki/Talk:O5m#Compression_Algorithms" class="">http://wiki.openstreetmap.org/wiki/Talk:O5m#Compression_Algorithms</a> o5m uses string tables specifically to avoid relying on general purpose compression. I find this unnecessary considering that zlib compression is quite effective, resource efficient (with adjustable compression level), and available practically everywhere.</div><div class=""><br class=""></div><div class="">There are a few other unusual design decisions documented and discussed at <a href="http://wiki.openstreetmap.org/wiki/O5m" class="">http://wiki.openstreetmap.org/wiki/O5m</a> and <a href="http://wiki.openstreetmap.org/wiki/Talk:O5m" class="">http://wiki.openstreetmap.org/wiki/Talk:O5m</a>. For example, strings are both introduced and terminated by a null byte, and are often stored in pairs (i.e. three null bytes per string pair, one of which is at the beginning of the string).</div><div class=""><br class=""></div><div class="">Of course I recognize o5m's contribution to the dialog on binary formats, and we can of course learn from the o5m concept, but my conclusion is that it does not have the combination of extreme simplicity and compactness necessary to complement the existing formats.</div><div class=""><br class=""></div><div class="">As for fixed sized blocks in vex, I did consider that option but couldn’t come up with a compelling reason for it. I can see the case for a maximum block size (so we know what the maximum size of allocation will be), but can you give a concrete example of how fixed-size blocks would be advantageous in practice? I would be very hesitant to split any entity across multiple blocks.</div><div class=""><br class=""></div><div class="">-Andrew</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 07 Feb 2016, at 09:06, Дмитрий Киселев <<a href="mailto:dmitry.v.kiselev@gmail.com" class="">dmitry.v.kiselev@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Looks pretty similar to o5m, except tags key=value are not round-buffered.<br class=""><br class=""></div>As a further extension, it would be nice to have the ability to have blocks of fixed size. <br class="">Just write nodes one by one while you haven't full-fill byte buffer.<br class=""></div><div class="">For extremely big relations (which are larger than one block) it's possible to mark two adjacent blocks as connected, but there should be a few of them.<br class=""></div><div class=""><br class=""></div>It would help to read write and seek over files.<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">2016-02-07 3:47 GMT+05:00 Stadin, Benjamin <span dir="ltr" class=""><<a href="mailto:Benjamin.Stadin@heidelberg-mobil.com" target="_blank" class="">Benjamin.Stadin@heidelberg-mobil.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto" class="">
<div class="">Hi Andrew,</div>
<div class=""><br class="">
</div>
<div class="">Cap'n Proto (successor of ProtoBuffer from the guy who invented ProtoBuffer) and FlatBuffers (another ProtoBuffer succesor, by Google) have gained a lot of traction since last year. Both eliminate many if the shortcomings of the original ProtoBuffer (allow
for random access, streaming,...), and improve on performance also.</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://github.com/google/flatbuffers" target="_blank" class="">https://github.com/google/flatbuffers</a></div>
<div class=""><br class="">
</div>
<div class="">Here is a comparison between ProtoBuffer competitors:</div>
<div class=""><a href="https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html" target="_blank" class="">https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html</a></div>
<div class=""><br class="">
</div>
<div class="">In my opinion FlatBuffers is the most interesting. It seems to have very good language and platform support, and has quite a high adoption rate already. </div>
<div class=""><br class="">
</div>
<div class="">I think that it's well worth to reconsider creating an own file format and parser for several reasons. Your concept looks well thought, it should be possible to implement a lighweight parser using FlatBuffers for your data scheme. </div>
<div class=""><br class="">
</div>
<div class="">Regards</div>
<div class="">Ben <br class="">
<br class="">
<div class="">Von meinem iPad gesendet</div>
</div><div class=""><div class="h5">
<div class=""><br class="">
Am 06.02.2016 um 22:37 schrieb Andrew Byrd <<a href="mailto:andrew@fastmail.net" target="_blank" class="">andrew@fastmail.net</a>>:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">Hello OSM developers,</div>
<div class=""><br class="">
</div>
<div class="">Last spring I posted an article discussing some shortcomings of the PBF format and proposing a simpler binary OSM interchange format called VEX. There was a generally positive response at the time, including helpful feedback from other developers.
Since then I have revised the VEX specification as well as our implementation, and Conveyal has been using this format in our own day-to-day work.</div>
<div class=""><br class="">
</div>
<div class="">I have written a new article describing of the revised format:<br class="">
</div>
<a href="http://conveyal.com/blog/2016/02/06/vex-format-part-two" target="_blank" class="">http://conveyal.com/blog/2016/02/06/vex-format-part-two</a>
<div class=""><br class="">
</div>
<div class="">
<div class="">The main differences are 1) it is more regular and even simpler to parse; and 2) file blocks are compressed individually, allowing parallel processing and seeking to specific entity types. It is no longer smaller than PBF, but still comparable
in size.</div>
<br class="">
</div>
<div class="">Again, I would welcome any comments you may have on the revised format and the potential for a shift to simpler binary OSM formats.</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class="">Andrew Byrd</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 29 Apr 2015, at 01:35, andrew byrd <<a href="mailto:andrew@fastmail.net" target="_blank" class="">andrew@fastmail.net</a>> wrote:</div>
<br class="">
<div class="">
<div class="">
<div class="">Hello OSM developers,<br class="">
</div>
<div class=""> </div>
<div class="">Over the last few years I have worked on several pieces of software that consume and produce the PBF format. I have always appreciated the advantages of PBF over XML for our use cases, but over time it became apparent to me that PBF is significantly
more complex than would be necessary to meet its objectives of speed and compactness.<br class="">
</div>
<div class=""> </div>
<div class="">Based on my observations about the effectiveness of various techniques used in PBF and other formats, I devised an alternative OSM representation that is consistently about 8% smaller than PBF but substantially simpler to encode and decode. This
work is presented in an article at <a href="http://conveyal.com/blog/2015/04/27/osm-formats/" target="_blank" class="">
http://conveyal.com/blog/2015/04/27/osm-formats/</a>. I welcome any comments you may have on this article or on the potential for a shift to simpler binary OSM formats.<br class="">
</div>
<div class=""> </div>
<div class="">Regards,<br class="">
</div>
<div class="">Andrew Byrd<br class="">
</div>
</div>
_______________________________________________<br class="">
dev mailing list<br class="">
<a href="mailto:dev@openstreetmap.org" target="_blank" class="">dev@openstreetmap.org</a><br class="">
<a href="https://lists.openstreetmap.org/listinfo/dev" target="_blank" class="">https://lists.openstreetmap.org/listinfo/dev</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><span class="">_______________________________________________</span><br class="">
<span class="">dev mailing list</span><br class="">
<span class=""><a href="mailto:dev@openstreetmap.org" target="_blank" class="">dev@openstreetmap.org</a></span><br class="">
<span class=""><a href="https://lists.openstreetmap.org/listinfo/dev" target="_blank" class="">https://lists.openstreetmap.org/listinfo/dev</a></span><br class="">
</div>
</blockquote>
</div></div></div>
<br class="">_______________________________________________<br class="">
dev mailing list<br class="">
<a href="mailto:dev@openstreetmap.org" class="">dev@openstreetmap.org</a><br class="">
<a href="https://lists.openstreetmap.org/listinfo/dev" rel="noreferrer" target="_blank" class="">https://lists.openstreetmap.org/listinfo/dev</a><br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature"><div dir="ltr" class="">Thank you for your time. Best regards.<br class="">Dmitry.</div></div>
</div>
</div></blockquote></div><br class=""></div></body></html>