[OSM-dev] Daily diff's retry

[Brett Henderson]

spaetz wrote:
> Reply to myself: It worked. We have now: http://planet.openstreetmap.org/2007-10-10--10-11-00:01:02.osc.bz2
>
> I intend to keep the last 7 daily osmosis changesets and delete older one. Changesets use the end time of the previous changeset as the starting point, so they should not miss out any changes if osmosis works as advertized.
>
> The file naming scheme is a little complicated and I take suggestions for better ones.
>
> Now people don't have a reason anymore to complain about too long intervals between updates; take the daily diffs and make your maps current!
>   
Nice work!  Thanks for the effort getting this going.

What is the start time for the first daily diff 
2007-10-10--10-10-09:57:01.osc.bz2?  If it includes changes prior to the 
time the current planet file was started we can use it now to create 
consistent daily planets.

If anybody is interested in using these daily diffs, the following 
information may be useful.  Due to the inconsistent dumps produced by 
the weekly planet process, some understanding of the diff process is needed.

The daily dump currently available can't be applied to a previous 
planet, any attempts to do so will probably appear to work but will 
result in a very inconsistent planet file with much potential for newly 
created elements to refer to other elements not yet existing (eg. new 
ways referring to nodes that were created after the planet was created 
but before the changeset interval).

To get a consistent snapshot at a point in time, it will be necessary to 
apply daily changesets to a planet created *after* the changeset and 
continue to apply daily changesets until the desired point in time is 
reached.  For example, the next planet will be 17th October.  If you 
pick the daily diffs for 16th-17th and 17th-18th and apply them 
sequentially you will have a consistent snapshot with no referential 
integrity issues containing all osm data as of 18th October.

Osmosis can be used to apply a diff in the following fashion.
osmosis --read-xml file="existingplanet.osm" --read-xml-change 
file="dailydiff.osc" --apply-change --write-xml file="newplanet.osm"

To work with compressed files:
osmosis --read-xml file="existingplanet.osm.bz2" compressionMethod=bzip2 
--read-xml-change file="dailydiff.osc.bz2" compressionMethod=bzip2 
--apply-change --write-xml file="newplanet.osm.bz2" compressionMethod=bzip2

To use the command line bzip2 for increased speed (on linux):
bzcat existingplanet.osm.bz2 | osmosis --read-xml file="/dev/stdin" 
--read-xml-change file="dailydiff.osc.bz2" compressionMethod=bzip2 
--apply-change --write-xml file="/dev/stdout" | bzip2 > newplanet.osm.bz2

To apply two changesets at once:
osmosis --read-xml file="existingplanet.osm" --read-xml-change 
file="dailydiff1.osc" --apply-change --read-xml-change 
file="dailydiff2.osc" --apply-change --write-xml file="newplanet.osm"

I've typed the above from memory so hopefully there aren't too many 
typos in there ...

One other thing is that the daily diffs include user information.  This 
might be interesting to some ...

Cheers,
Brett