[OSM-dev] Hourly diffs are missing edits (too)

[Peter Körner]

John Smith schrieb:
> 2009/10/7 Lennard <ldp at xs4all.nl>:
>> John Smith wrote:
>>
>>> I'm trying to work out how to know how many diffs per day there are so
>>> that can be used to figure out how to keep up to date when things
>>> flick over between days etc.
>> The state.txt files hold a timestamp field, so see if that's all you
>> need for your needs.
> 
> That gives you the last file, I'm just asking about the directory structure...

Each diff has a simple numeric id, that is incremented by one each time 
a diff is created. [1] gives you the id of the last diff created, 
acually it's 42957. This number is padded with zeros to the left until 
it's 9 digits long: 000042957. Then it's split at each three digits: 
000/042/957. The last diff is .../000/042/958.osc.gz [2].

So what you need to do to keep an up2date db the procedure could be sth. 
like this:

  1. download a planet-dump, e.g. planet-091007.osm.bz2 which is of
     09-Oct-2009 03:37

  2. fetch the id of the last diff on the servers (remote_last) by
     looking at [1]

  3. count from 0 to remote_last and for each loop
  3.1. fetch the state-file and check the timestamp against the
       creation date of the planet-dump
  3.2. if they are close enough you got the diff-id you should start with
  3.3. save this id in local_last

  4. start the update loop
  4.1 fetch the id of the last diff on the servers and save it to
      remote_last.
  4.2 If it's > then local_last count from local_last up to remote_last
      and for each id
  4.2.1 fetch the osc-file and try to merge it into the db using osmosis
  4.3 sleep a minute


If your server is going to sleep and you want to restart some time late, 
just restore local_last e.g. from disk (this is the last diff id you 
imported successfully and restart with 4.)

I never did sth. like this but this is what i think the diffs are meant 
to be used.

Peter


[1] http://planet.openstreetmap.org/minute-replicate/state.txt
[2] http://planet.openstreetmap.org/minute-replicate/000/042/959.osc.gz