[OSM-dev] visible-Flag in PBF

Christian Vetter veaac.fdirct at gmail.com
Sat May 7 20:06:18 BST 2011


On Sat, May 7, 2011 at 5:46 AM, Scott Crosby <scott at sacrosby.com> wrote:
> I don't see a need to define 4 new blob-types as PrimitiveBlocks messages
> already contain all of those types. Just one new type
> OSMInvisibleHistoryData. What the
> motivation otherwise?

You are right, of course.

> One idea: Each BlobHeader includes an 'indexdata' field. We could
> store a protocol buffer message there. It is available without parsing
> the blob. It needs to be kept small, but adding something like:
> message IndexData {
>   optional int32 nodecount = 16
>   optional int32 waycount = 17
>   optional int32 relationcount = 18
> }

I guess this would work. However, you still would have one random file
access for each blob. A normal hard drive has a seek time of about 5ms
-> skipping 1000 blobs would take about 5 seconds.

Would this still make it possible to add further index data structures later on?

> Can you and others who want this feature please make a convincing case
> of why every planet should get about 1mb bigger?

I would use this for parsing a set of relations / ways without using a
lot of memory / disk space: First scan relations, then scan ways, then
scan nodes. Adding one megabyte does not seem much, especially since
we already have about the same amount of redundancy in the blob
headers by using a string for the blob type. We could decrease the
overhead a little bit by just storing a bool for each of the three.

However, the same could also be achieved by having the PBF elements
ordered such that relations come before ways and ways come before

> DenseNodes make sense for two reasons. They reduce the protobuf header
> overheads (of about 30% before compression), and they allow delta
> coding for id&lat&lon. Nodes also account for >90% of the entities and
> most of them have no tags. Ways and relations are individually much
> larger, so header overheads matter less, and I already delta compress
> their member id's.

Ok, makes sense.

> This is already documented in the Wiki. (See OSMHeader section). The
> current osmosis writer doesn't set that feature because AFAICT, there
> is no functionality in osmosis for the writer to know that the file is
> sorted so that it knows to set that flag.

I have no idea how the PBF extracts are actually generated from the
OSM data base. Would there be some way to influence the order of
elements and letting Osmosis know about this without adding much


Christian Vetter

More information about the dev mailing list