[OSM-dev] visible-Flag in PBF

Christian Vetter veaac.fdirct at gmail.com
Thu May 5 16:05:46 BST 2011


Hi,

As far as I understand Google's protobuffer it should be possible to
add new ( optional ) fields. If a reader does not know about them they
are just skipped. It is also possible to remove optional fields. You
have to take care with the IDs you assign, though.

Therefore you have at least two possibilities:
A)
   Update the PBF specifications to include new optional fields for
visibility. If you do this right, the generated PBF will be readable
by all previous PBF readers.
B)
   Add an new optional_feature ( e.g. "VisibilityBlocks" ) and a new
type of blob. This blob would contain all the visible flags of the
previous PBF block and the file would look somewhat like this:
   PBFHeader PBFBlock VisibilityBlock PBFBlock VisibilityBlock...
   All complient PBF readers should still be able to parse this file
since they are obliged to skip blocks they do not know ( but not all
readers are standard compliant yet, e.g., MoNav supports it only the (
unstable ) 0.3.1 version ).
   You only have to release the protobuffer information of your
visibility blocks in this case for readers to be able to read the
visibility data.

Overall solution A might look cleaner, since the data is directly
associated with the objects, which makes streaming easier. However a
reader supporting this data will have to be modified in several
places.
I would prefer solution B. Less modifications to the coder would be
necessary to read that format. Streaming is still possible, you only
have to perform a one-block read-ahead. And all existing readers can
skip visibility information really fast. Last, it does not require you
to modify the PBF format, only to release the VisibilityBlock
protobuffer definitions.

Regards,

Christian

On Thu, May 5, 2011 at 4:42 PM, Peter Körner <osm-lists at mazdermind.de> wrote:
> Hi
>
> I'm working heavily with the full-history-dumps. Lately I wrote a splitter
> [1] that allows splitting the full-history-dump into smaller extracts and I
> plan to publish some of the extracts on a GWDG mirror.
>
> Right now the splitter uses osmium as Framework, because its xml-reader
> passes the visible-flag along through the pipeline. The xml-writer I wrote
> for osmium [2] is also able to write the visible-flag out.
>
> But actually it would be nice to let the whole xml-bz2-thing be and use our
> new cool pbf stuff. But unfortunately the pbf format does not support saving
> the visible-flag, which is really important when working with the history.
>
> I talked with Jochen Topf about the extensibility of out pbf-format and he
> anticipated problems with the DenseInfo message. So my questions are:
>  - is it possible for an app like my splitter to add a boolean
>   visible-flag to nodes, ways and relations in a pbf file, so that
>   other apps like osmosis can still read the generated pbf file (just
>   ignoring the visible flag)
>
>  - if not: is it feasible to release a new version of the pbf
>   definition, that contains a boolean visible-flag
>
>  - if not: how could a pbf-file generated with the visible-flag but
>   incompatible with common apps like osmosis be named, to clearly show
>   that it's NO .osm.pbf file? (eg. germany.hosm.pbf or
>   germany.osm.history.pbf or ..)
>
> Peter
>
>
> [1]
> <https://github.com/MaZderMind/OpenStreetMap-History-API/tree/master/splitter>
> [2] <https://github.com/MaZderMind/osmium/tree/xml_output>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>



More information about the dev mailing list