[OSM-dev] Simpler binary OSM formats

Stadin, Benjamin Benjamin.Stadin at heidelberg-mobil.com
Mon Feb 8 08:48:36 UTC 2016


> When you need relations-ways-nodes read order, blocks will save you  from unnecessary read-through the whole file (yes, you can skip decompression for nodes/ways but still you must read the whole file).

Or use mmap. Or directly Cap’n Proto or FlatBuffers, which support this out of the box.
https://capnproto.org/cxx.html#using-mmap

> Second example: find something by id, if you have blocks it's easy to map whole block into memory and do a binary search for logN block reads instead of seeing through a file all the time.

I believe it will make implementation unnecessary more complicated. In case of Cap’n Proto you could set the initial position of your binary search to the array index, under the hood it would use mmap to seek the location. Also you can improve on lookup: for example start a binary search for way with id 999 at ways[999] instead of ways[waysCount/2]

Von: Дмитрий Киселев <dmitry.v.kiselev at gmail.com<mailto:dmitry.v.kiselev at gmail.com>>
Datum: Sonntag, 7. Februar 2016 um 20:10
An: Andrew Byrd <andrew at fastmail.net<mailto:andrew at fastmail.net>>
Cc: Benjamin Stadin <benjamin.stadin at heidelberg-mobil.com<mailto:benjamin.stadin at heidelberg-mobil.com>>, "dev at openstreetmap.org<mailto:dev at openstreetmap.org>" <dev at openstreetmap.org<mailto:dev at openstreetmap.org>>
Betreff: Re: [OSM-dev] Simpler binary OSM formats

As for fixed sized blocks in vex, I did consider that option but couldn’t come up with a compelling reason for it. I can see the case for a maximum block size (so we know what the maximum size of allocation will be), but can you give a concrete example of how fixed-size blocks would be advantageous in practice? I would be very hesitant to split any entity across multiple blocks.
[https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif]

When you need relations-ways-nodes read order, blocks will save you  from unnecessary read-through the whole file (yes, you can skip decompression for nodes/ways but still you must read the whole file).

Second example: find something by id, if you have blocks it's easy to map whole block into memory and do a binary search for logN block reads instead of seeing through a file all the time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20160208/6a8b1562/attachment.html>


More information about the dev mailing list