[OSM-talk] New Daily Diff Files

[Brett Henderson]

Hi All,

Apologies for cross-posting to two lists but discussions have been split 
across both lists.  Please reply to the dev list.

There are now some new daily diff files available on the planet server 
that address some issues with the existing daily diff files.  These new 
files are produced using a more reliable mechanism that is already used 
by the hourly and minute diffs.  They have a different naming standard 
so there's no conflict.
http://planet.openstreetmap.org/daily/

The timestamp.txt file will tell you what the latest file is available 
to be downloaded rather than relying on 404 server responses if new 
files aren't available.  The osmosis --read-change-interval task will 
work with the new files but not the old.  The task will merge all 
available files into a single change stream that can then be written to 
a file using --write-xml-change (for subsequent import to a db) or 
passed to another task such as --apply-change for merging into existing 
xml files.

The new files are gzip compressed due to performance issues with bzip2 
compression in java.  This means they're bigger but we're still only 
talking approximately 10MB per day.

There is one big GOTCHA.
The new files use UTC timing, the old files use BST timing.  This means 
that the contents of the files are different.  If you transition to the 
new file format you should re-apply the new gzip file corresponding to 
the most recently applied bzip2 file in order to capture the missing 
hour.  If you don't do this I *think* you'll miss an hours worth of data 
from 11pm to midnight.  If the last file you imported was 
daily-20080622-20080623.osc.bz2, you should start on the new files from 
20080622-20080623.osc.gz.

Let me know if you see any problems.

Cheers,
Brett