[OSM-dev] Incomplete diffs?

[marqqs at gmx.eu]

Hello,

meanwhile osmupdate has been changed to download replication diffs only.

But there is still an issue I cannot explain...

I thought that the "normal" daily diffs would lack of some data which were provided around midnight, but there seem data to vanish from the middle of the day too:

# normal diff
$ zcat 20111103-20111104.osc.gz |grep -c "timestamp=\"2011-11-03T12:"
58968

# replication diff
$ cat 1103-1104.osc |grep -c "timestamp=\"2011-11-03T12:"
59068

And yes, I thought on cumulating the version in the second file before I started counting with grep.

Of course, it could easily be that I overlooked something - it's late in Germany. :-)
Does anyone have an idea?

Markus


-------- Original-Nachricht --------
> Datum: Sun, 06 Nov 2011 00:32:49 +0100
> Von: marqqs at gmx.eu
> An: Frederik Ramm <frederik at remote.org>, dev at openstreetmap.org
> Betreff: Re: [OSM-dev] Incomplete diffs?

> Hi Frederik,
> 
> thanks for the explanation!
> 
> The _replication_ diffs are the right choice if you want to update a full
> history file.
> 
> Most people who update their OSM files on a regular basis do not need
> replication diffs, they are satisfied with the newest version of each object
> which has been changed.
> 
> > So, if you want to use daily diffs but avoid the danger of missing 
> > edits, use the replication diff.
> 
> Very good advice.
> 
> Until today I chose NOT to use the diffs in the
> planet.openstreetmap.org/history/ directory because they are outdated. They usually come with a delay
> of 25 hours. Du you know if the creation process could be accelerated
> somehow?
> 
> Now I will attend to osmupdate and try to change from daily normal diffs
> to daily replication diffs. Seems to be better than loosing objects once in
> a while.
> 
> Meanwhile people can use Osmosis, or run osmupdate with the --hourly
> option which will restrict the program to replication diffs.
> 
> Markus
> 
> -------- Original-Nachricht --------
> > Datum: Sat, 05 Nov 2011 22:56:01 +0100
> > Von: Frederik Ramm <frederik at remote.org>
> > An: dev at openstreetmap.org
> > Betreff: Re: [OSM-dev] Incomplete diffs?
> 
> > Hi,
> > 
> > On 11/05/2011 06:58 PM, marqqs at gmx.eu wrote:
> > > Meanwhile I found out that this node simply did not appear in the
> daily
> > diffs:
> > > http://www.openstreetmap.org/browse/node/1470178889
> > >
> > > It was crated at 2011-10-16T23:58Z by a large changeset along with
> > 23.000 other nodes.
> > > Neither the 16/17 nor the 17/18 daily diff contain this node whereas
> the
> > hourly diff from October 17 01:00 does.
> > 
> > There are two types of diffs; "replication diffs" and normal diffs. A 
> > replication diff contains everything that happened between two 
> > timestamps, including multiple changes of the same object, whereas a 
> > normal diff only contains the information required to get from state 1 
> > to state 2.
> > 
> > Also, replication diffs are created in a relatively fail-safe process 
> > with Osmosis whereas the normal diffs can miss changes in some cases 
> > when a long-running database transaction that was created before 0:00 
> > extends past the time when the diff is created. (There was a time when 
> > we had only "normal" diffs, and it was near impossible to make sure the 
> > minutely/hourly ones did not miss anything.)
> > 
> > For minutely and hourly diffs, we only offer replication diffs these 
> > days. For daily diffs, we have the normal ones under 
> > planet.openstreetmap.org/daily, as well as the replication diffs under 
> > planet.openstreetmap.org/history.
> > 
> > The normal diff indeed lacks the node in question, but the daily 
> > replication diff under history/2011/1016-1017.osc.gz has it.
> > 
> > So, if you want to use daily diffs but avoid the danger of missing 
> > edits, use the replication diff.
> > 
> > Frankly I don't know why the normal daily diffs are still created at 
> > all; if one really wanted to offer a reduced-traffic version of the 
> > replication diffs then it would indeed make sense to simply deflate the 
> > replication diff using Osmosis' --simplify-change task.
> > 
> > Bye
> > Frederik
> > 
> > -- 
> > Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09"
> E008°23'33"
> > 
> > _______________________________________________
> > dev mailing list
> > dev at openstreetmap.org
> > http://lists.openstreetmap.org/listinfo/dev
> 
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev