[OSM-dev] Timestamp in PBF files

[marqqs at gmx.eu]

Hi Scott,

in brief to the 1-degrees granularity:

1. Do whole processing in 64 bit:
This would mean to need much more RAM space when processing ways' coordinates. We should not do this unless this granularity is really required.

2. Your formula:
  latitude_int = ((lat_offset + granularity*lat)/50+1)/2
Good idea, but again, this would mean one more multiplication, one more division (and two additions, one shift). These operations usually can be done in no time, however that's different if you need to do them a Billion times.
There are still people out there who have 32 bit machines, I presume they do not have 64 bits hardware multiplication units, hence the processing time will increase.

3. Process sequence:
Using the granularity factor, lon/lat of every node in an OSMData fileblock must be read, stored temporarily and transformed later. Thus you have to access every data twice: first to read it, and a second time when you transform its granularity. This might be a flaw in PBF data model... Could we at least change this in that manner that the granularity information comes _before_ the real data? Same applies to lon/lat offset and date granularity.

In the end - there always will be a lot of programs which do not need this quasi "optional feature" "granularity" and simply will not support it.


Metadata...

We had the same discussion a year ago. Do you remember?
https://wiki.openstreetmap.org/wiki/Talk:PBF_Format#File_Timestamp.3F
I'm curious if - and I hope that - we manage to extend the PBF data format this time. :-)

The file time stamp I added was meant as an interim solution: I took the already defined "optional feature" and stored a key-val pair in it, for example "timestamp=2011-10-16T15:45:00Z".

I think this example shows what we really need: a flexible format for file related meta data. With key-val pairs, everyone could add optional data whenever they are needed in a toolchain. This is the flexibility we are used to have from OSM XML format.

To combine this flexibility with the advantages of Protobuf format (compressed storage of different data types) we need to allow meta formatted objects - or something like this:

message HeaderBlock {
  ...
  repeated HeaderMeta = 20;
}

message HeaderMeta {
  required string HeaderKey = 1;
  optional HeaderMetaVarint = 10;
  optional HeaderMetaString = 12;
// see type definitions there: https://wiki.openstreetmap.org/wiki/PBF#Format_example
// Only _one_ of the three optional objects should be used; did not know how to define this in Protobuf without an additional hierarchy layer.
}

What do you think about this suggestion?

Markus