[OSM-dev] Incomplete diffs?

marqqs at gmx.eu marqqs at gmx.eu
Sat Nov 5 21:33:26 GMT 2011


Hi Erik,

thanks for your help. The missing node seems to be available via minutely and hourly diff files but NOT via the daily file.

Meanwhile I found an explanation in the Wiki:
http://wiki.openstreetmap.org/wiki/Planet.osm/diffs

"The data in the file is the change in data between midnight on the first and second days, as identified by the timestamps on the current data. Because of the delay in creating the file it is very unlikely, but possible, that some data may be missing."

If the problem is already known, why is the file not created with overlapping time borders, let's say from 00:00 to 24:01? This would be better than loosing data.

Alternatively, the daily diff could easily be created by merging the latest 24 hourly diffs.
(Just tried this, takes 10 seconds on my medium server.)

Maybe we should propose to change the toolchain...?

Markus

-------- Original-Nachricht --------
> Datum: Sat, 5 Nov 2011 21:06:30 +0100
> Von: Erik Johansson <emj at kth.se>
> An: marqqs at gmx.eu
> Betreff: Re: [OSM-dev] Incomplete diffs?

> On Sat, Nov 5, 2011 at 18:58,  <marqqs at gmx.eu> wrote:
> > Hi all,
> >
> > this week, a user of osmupdate and osmfilter asked me for help. The
> filter program's output contained a certain way but not all of its nodes. This
> must not happen, of course, and I thought this would be a program error at
> first.
> >
> > Meanwhile I found out that this node simply did not appear in the daily
> diffs:
> > http://www.openstreetmap.org/browse/node/1470178889
> >
> > It was crated at 2011-10-16T23:58Z by a large changeset along with
> 23.000 other nodes.
> > Neither the 16/17 nor the 17/18 daily diff contain this node whereas the
> hourly diff from October 17 01:00 does.
> >
> > How often does this happen? Is there a statistic about such gaps?
> >
> > Should I better switch to hourly diffs instead of daily ones? This would
> increase the data traffic a bit as hourly diffs do not cumulate multiple
> versions for each object.
> >
> 
> Just for reference I can't find that node in these dailies either:
> http://planet.openstreetmap.org/daily/20111015-20111016.osc.gz
> http://planet.openstreetmap.org/daily/20111016-20111017.osc.gz
> http://planet.openstreetmap.org/daily/20111017-20111018.osc.gz
> 
> But it's available in the minutely:
> http://planet.openstreetmap.org/minute-replicate/001/028/610.gz
> 
> 
> Somone on IRC just said this the other day:
> > Weekly I think are still done with the planetdiff program
> > everything else is done with osmosis [by merge from minutely].
> 
> I don't know if this is correct but try to merge two or three hours of
> minutely with Osmosis, and see what you get.
> 
> /Erik



More information about the dev mailing list