[OSM-dev] Incomplete diffs?
marqqs at gmx.eu
marqqs at gmx.eu
Mon Nov 7 01:24:16 GMT 2011
Hello,
meanwhile osmupdate has been changed to download replication diffs only.
But there is still an issue I cannot explain...
I thought that the "normal" daily diffs would lack of some data which were provided around midnight, but there seem data to vanish from the middle of the day too:
# normal diff
$ zcat 20111103-20111104.osc.gz |grep -c "timestamp=\"2011-11-03T12:"
58968
# replication diff
$ cat 1103-1104.osc |grep -c "timestamp=\"2011-11-03T12:"
59068
And yes, I thought on cumulating the version in the second file before I started counting with grep.
Of course, it could easily be that I overlooked something - it's late in Germany. :-)
Does anyone have an idea?
Markus
-------- Original-Nachricht --------
> Datum: Sun, 06 Nov 2011 00:32:49 +0100
> Von: marqqs at gmx.eu
> An: Frederik Ramm <frederik at remote.org>, dev at openstreetmap.org
> Betreff: Re: [OSM-dev] Incomplete diffs?
> Hi Frederik,
>
> thanks for the explanation!
>
> The _replication_ diffs are the right choice if you want to update a full
> history file.
>
> Most people who update their OSM files on a regular basis do not need
> replication diffs, they are satisfied with the newest version of each object
> which has been changed.
>
> > So, if you want to use daily diffs but avoid the danger of missing
> > edits, use the replication diff.
>
> Very good advice.
>
> Until today I chose NOT to use the diffs in the
> planet.openstreetmap.org/history/ directory because they are outdated. They usually come with a delay
> of 25 hours. Du you know if the creation process could be accelerated
> somehow?
>
> Now I will attend to osmupdate and try to change from daily normal diffs
> to daily replication diffs. Seems to be better than loosing objects once in
> a while.
>
> Meanwhile people can use Osmosis, or run osmupdate with the --hourly
> option which will restrict the program to replication diffs.
>
> Markus
>
> -------- Original-Nachricht --------
> > Datum: Sat, 05 Nov 2011 22:56:01 +0100
> > Von: Frederik Ramm <frederik at remote.org>
> > An: dev at openstreetmap.org
> > Betreff: Re: [OSM-dev] Incomplete diffs?
>
> > Hi,
> >
> > On 11/05/2011 06:58 PM, marqqs at gmx.eu wrote:
> > > Meanwhile I found out that this node simply did not appear in the
> daily
> > diffs:
> > > http://www.openstreetmap.org/browse/node/1470178889
> > >
> > > It was crated at 2011-10-16T23:58Z by a large changeset along with
> > 23.000 other nodes.
> > > Neither the 16/17 nor the 17/18 daily diff contain this node whereas
> the
> > hourly diff from October 17 01:00 does.
> >
> > There are two types of diffs; "replication diffs" and normal diffs. A
> > replication diff contains everything that happened between two
> > timestamps, including multiple changes of the same object, whereas a
> > normal diff only contains the information required to get from state 1
> > to state 2.
> >
> > Also, replication diffs are created in a relatively fail-safe process
> > with Osmosis whereas the normal diffs can miss changes in some cases
> > when a long-running database transaction that was created before 0:00
> > extends past the time when the diff is created. (There was a time when
> > we had only "normal" diffs, and it was near impossible to make sure the
> > minutely/hourly ones did not miss anything.)
> >
> > For minutely and hourly diffs, we only offer replication diffs these
> > days. For daily diffs, we have the normal ones under
> > planet.openstreetmap.org/daily, as well as the replication diffs under
> > planet.openstreetmap.org/history.
> >
> > The normal diff indeed lacks the node in question, but the daily
> > replication diff under history/2011/1016-1017.osc.gz has it.
> >
> > So, if you want to use daily diffs but avoid the danger of missing
> > edits, use the replication diff.
> >
> > Frankly I don't know why the normal daily diffs are still created at
> > all; if one really wanted to offer a reduced-traffic version of the
> > replication diffs then it would indeed make sense to simply deflate the
> > replication diff using Osmosis' --simplify-change task.
> >
> > Bye
> > Frederik
> >
> > --
> > Frederik Ramm ## eMail frederik at remote.org ## N49°00'09"
> E008°23'33"
> >
> > _______________________________________________
> > dev mailing list
> > dev at openstreetmap.org
> > http://lists.openstreetmap.org/listinfo/dev
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
More information about the dev
mailing list