[OSM-dev] change replication and replication lag

[mmd]

On 3/10/21 11:05 AM, Roland Olbricht wrote:
> Am 10.03.21 um 10:29 schrieb Jochen Topf via dev:
>>>> What are others using? switch2osm still refers to a mod_tile supplied
>>>> script using osmose.
> [..]
>>> osmium fileinfo --extended --get=data.timestamp.last
>>
>> That has the advantage of definitly being correct, because it looks at
>> the actual data and not some state file which might or might not be
>> correct.
> 
> Please note the pitfall that later changefiles may contain older unseen
> changes, i.e. the date ranges overlap. E.g.
> 
> If there is an object with timestamp 2021-03-09T22:14:57Z in
> https://planet.openstreetmap.org/replication/minute/004/448/577.osc.gz
> then it absolutely can happen that in
> https://planet.openstreetmap.org/replication/minute/004/448/578.osc.gz
> there is another object with timestamp half an hour or so earlier.
> 


Back in 2016/2017, we did indeed have changeset uploads taking 30
minutes, sometimes even more than two hours (see [1] for a detailed
analysis).

Since that changeset upload process moved to cgimap a couple of years
ago, those extreme runtimes should no longer occur. Today, we're more
likely experiencing an overlap in a range of maybe 5-20s for really
large changesets.

Nevertheless, the underlying issue Roland outlined earlier on still
applies. There's some more detailed explanation on the Osmosis OSM wiki
page', which explains the differences between a transaction-aligned vs.
time-aligned replication, in case you're interested.

[1]
https://github.com/openstreetmap/openstreetmap-website/issues/1710#issuecomment-353631926