[OSM-dev] Hourly diffs are missing edits (too)

John Smith deltafoxtrot256 at gmail.com
Tue Oct 13 13:50:40 BST 2009


2009/10/10 Peter Körner <osm-lists at mazdermind.de>:
>
>
> John Smith schrieb:
>> 2009/10/7 Lennard <ldp at xs4all.nl>:
>>> John Smith wrote:
>>>
>>>> I'm trying to work out how to know how many diffs per day there are so
>>>> that can be used to figure out how to keep up to date when things
>>>> flick over between days etc.
>>> The state.txt files hold a timestamp field, so see if that's all you
>>> need for your needs.
>>
>> That gives you the last file, I'm just asking about the directory structure...
>
> Each diff has a simple numeric id, that is incremented by one each time
> a diff is created. [1] gives you the id of the last diff created,
> acually it's 42957. This number is padded with zeros to the left until
> it's 9 digits long: 000042957. Then it's split at each three digits:
> 000/042/957. The last diff is .../000/042/958.osc.gz [2].
>
> So what you need to do to keep an up2date db the procedure could be sth.
> like this:
>
>  1. download a planet-dump, e.g. planet-091007.osm.bz2 which is of
>     09-Oct-2009 03:37
>
>  2. fetch the id of the last diff on the servers (remote_last) by
>     looking at [1]
>
>  3. count from 0 to remote_last and for each loop
>  3.1. fetch the state-file and check the timestamp against the
>       creation date of the planet-dump
>  3.2. if they are close enough you got the diff-id you should start with
>  3.3. save this id in local_last
>
>  4. start the update loop
>  4.1 fetch the id of the last diff on the servers and save it to
>      remote_last.
>  4.2 If it's > then local_last count from local_last up to remote_last
>      and for each id
>  4.2.1 fetch the osc-file and try to merge it into the db using osmosis
>  4.3 sleep a minute
>
>
> If your server is going to sleep and you want to restart some time late,
> just restore local_last e.g. from disk (this is the last diff id you
> imported successfully and restart with 4.)
>
> I never did sth. like this but this is what i think the diffs are meant
> to be used.
>
> Peter
>
>
> [1] http://planet.openstreetmap.org/minute-replicate/state.txt
> [2] http://planet.openstreetmap.org/minute-replicate/000/042/959.osc.gz
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>

Thanks, I've since implemented this, what was confusing was the use of
the word minutely, when the filenames don't refer to particular
minutes.




More information about the dev mailing list