[OSM-dev] visible-Flag in PBF

Scott Crosby scott at sacrosby.com
Sat May 7 02:58:07 BST 2011

On Thu, May 5, 2011 at 9:42 AM, Peter Körner <osm-lists at mazdermind.de>
> Hi
> I'm working heavily with the full-history-dumps. Lately I wrote a splitter
> [1] that allows splitting the full-history-dump into smaller extracts and
> plan to publish some of the extracts on a GWDG mirror.
> Right now the splitter uses osmium as Framework, because its xml-reader
> passes the visible-flag along through the pipeline. The xml-writer I wrote
> for osmium [2] is also able to write the visible-flag out.
> But actually it would be nice to let the whole xml-bz2-thing be and use
> new cool pbf stuff. But unfortunately the pbf format does not support
> the visible-flag, which is really important when working with the history.

There are two implementation approaches.

> I talked with Jochen Topf about the extensibility of out pbf-format and he
> anticipated problems with the DenseInfo message. So my questions are:

>  - is it feasible to release a new version of the pbf
>   definition, that contains a boolean visible-flag

Yes. If you want to create an incompatable hpbf (History PBF) format that is
very similar to the existing PBF format. Encode the visible flag into the
protocol buffers along the lines of Jochen's suggested patch, with a few
small tweaks. Existing pbf code can support the new format with very minor
changes. This would be an incompatible change because it is impossible to
telling existing software to ignore nodes whose visibility is 'no'. They're
there, and will be read. You'll need to indicate these files with a new
required_features = "InvisibleFlag", and conforming software will reject the

>  - is it possible for an app like my splitter to add a boolean
>   visible-flag to nodes, ways and relations in a pbf file, so that
>   other apps like osmosis can still read the generated pbf file (just
>   ignoring the visible flag)

Yes, but via a different mechanism. Create a new blob type:
'OSMInvisibleHistoryData', it will contain a serialized ordinary
PrimitiveBlock message. In this block you'll place invisible
nodes/ways/relations. Standard readers will see this block and skip right
past it. Your reader will feed it to the normal pbf decoder function and
have it mark those entities as invisible. You'll want to indicate these
files with a new optional_feature 'ContainsInvisibleHistoryData'.

>  - if not: how could a pbf-file generated with the visible-flag but
>   incompatible with common apps like osmosis be named, to clearly show
>   that it's NO .osm.pbf file? (eg. germany.hosm.pbf or
>   germany.osm.history.pbf or ..)

If you go with the incompatible approach, I'd name it .hpbf.

Which approach do you want? Or both?

Also, if we are going to do another revision of the format, I will use it as
an opportunity to change from java package crosby.binary to osm.format.pbf.

And maybe add other features? CRC32?

What else do people want to consider putting in? Anything else that would
help in recording history? Consider adding LZMA? Someone want to run a test?


> Peter
> [1]
> <
> [2] <https://github.com/MaZderMind/osmium/tree/xml_output>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20110506/c2ea649f/attachment-0001.html>

More information about the dev mailing list