[OSM-dev] Timestamp in PBF files

Brett Henderson brett at bretth.com
Fri Nov 23 11:24:15 GMT 2012


On 21 November 2012 19:43, Frederik Ramm <frederik at remote.org> wrote:

> Hi,

<snip>

>
> To be self-contained, it should be sufficient to include the "baseURL"
> from configuration.txt, no?
>
> So maybe:
>
>
>   optional string writingprogram = 16;
>   optional string source = 17;
>   optional sint64 timestamp = 18;
>   optional sint64 replication_timestamp = 19;
>   optional string replication_url = 20;
>
> I don't know if the sequenceNumber from state.txt adds any value, if it
> does then one could throw that in as well.


I've been explicitly cc'd on the original message so I should put in an
appearance ;-)

*If* this information is intended to be used as an input into replication
processes then the sequence number is essential.  Osmosis writes a
timestamp in the state.txt file, but it only for identifying the right
sequence number to begin replication with.  All replication processing
requires the sequence number.  Attempting to use a timestamp is
theoretically possible but it's much less efficient and not how it was
supposed to work.

However, utilising this new sequence number in Osmosis will require some
significant changes.  The current task that figures out what changes to
download (ie. --read-replication-interval) is totally independent of the
task that applies changes to a snapshot (ie. --apply-change).  The simplest
solution would be to write an "uber" task that is specifically aimed at
patching planet files, but it will be an all-in-one task that can't be
combined with others.  It *may* be possible to modify pipeline
initialisation to allow all tasks to synchronise replication numbers before
beginning processing, but that will be a lot more complicated.

Updating the timestamp and sequence number after processing will also
require some changes because it impacts a number of tasks.  All tasks will
have to propagate the field (shouldn't be too difficult), but tasks such as
--apply-change will need to be smart about which input source they use as
the source of truth for the sequence number.  It's all possible, but not a
trivial change.

Perhaps this is a non-issue if everybody uses osmupdate these days anyway
:-)

As for the PBF format itself, I don't have any opinions.  I'm more than
happy for those who are more familiar with it to come up with a solution.
I'll do my best to accommodate it.

Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20121123/e09a2366/attachment.html>


More information about the dev mailing list