[OSM-dev] visible-Flag in PBF

Jochen Topf jochen at remote.org
Fri May 6 09:00:51 BST 2011

On Thu, May 05, 2011 at 05:05:46PM +0200, Christian Vetter wrote:
> As far as I understand Google's protobuffer it should be possible to
> add new ( optional ) fields. If a reader does not know about them they
> are just skipped. It is also possible to remove optional fields. You
> have to take care with the IDs you assign, though.

Unfortunately there is a problem in this specific case if I understand
the protobuffer doc correctly:

These are the canges we would need to make:

--- a/src/osmformat.proto
+++ b/src/osmformat.proto
@@ -126,6 +126,7 @@ message Info {
    optional int64 changeset = 3;
    optional int32 uid = 4;
    optional uint32 user_sid = 5; // String IDs
+   optional bool visible = 6 [default = true];
 /** Optional metadata that may be included into each primitive. Special dense format used in DenseNodes. */
@@ -135,6 +136,7 @@ message DenseInfo {
    repeated sint64 changeset = 3 [packed = true]; // DELTA coded
    repeated sint32 uid = 4 [packed = true]; // DELTA coded
    repeated sint32 user_sid = 5 [packed = true]; // String IDs for usernames. DELTA coded
+   repeated bool visible = 6 [packed = true];

The first one is unproblematic. It should add an optional field with
a default value. All old code should still work. The Info message
is used for encoding nodes, ways, and relations.

But there is a different encoding for DenseNodes which is much more efficient
than the simple Node encoding. The DenseNodes is what everybody is using. And
that uses the DenseInfo message. The DenseInfo uses repeated fields which
can't be optional.

> Therefore you have at least two possibilities:
> A)
>    Update the PBF specifications to include new optional fields for
> visibility. If you do this right, the generated PBF will be readable
> by all previous PBF readers.

Yes, that would be very nice. But I don't see at the moment how to do that
in the DenseNode case.

> B)
>    Add an new optional_feature ( e.g. "VisibilityBlocks" ) and a new
> type of blob. This blob would contain all the visible flags of the
> previous PBF block and the file would look somewhat like this:
>    PBFHeader PBFBlock VisibilityBlock PBFBlock VisibilityBlock...
>    All complient PBF readers should still be able to parse this file
> since they are obliged to skip blocks they do not know ( but not all
> readers are standard compliant yet, e.g., MoNav supports it only the (
> unstable ) 0.3.1 version ).

I don't like this solution, because it removes streaming capability. You have
to read thousands of objects and store them, then look whether a
VisibilityBlock comes after the PBFBlock, before you can do something with
those objects. Or you have to store those two blocks and read them object by
object merging the information. Yuck.

At the moment I only see the option of having the DenseInfoWithVisible message
that supersedes the DenseInfo message and note it in the header that we need
this capability. But I am hoping there is a better solution that doesn't lead
to an incompatible change.

Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298

More information about the dev mailing list