[OSM-dev] New tool to generate incremental planet.osm updates (planetdiff/planetpatch)

Jon Burgess jburgess777 at googlemail.com
Fri Apr 6 14:05:43 BST 2007


On Fri, 2007-04-06 at 14:38 +0200, Andreas Volz wrote:
> Am Fri, 06 Apr 2007 12:34:17 +0100 schrieb Jon Burgess:
> 
> > Hi all,
> > 	I've developed a diff/patch tool for handling planet.osm dump
> > files. This allows an incremental diff to be generated between 2
> > planet.osm dumps. This diff file can then be applied on top of the
> > old planet.osm to generate the new one. 
> > 
> > If we published these diff files on planet.openstreetmap.org then it
> > could make the downloads to obtain a new planet.osm dump much
> > smaller. I think the typical weekly diff file is in the order of 5MB
> > to 10MB when bz2 compressed.
> > 
> > 
> > The usage summary is:
> > 
> > Patch creation:
> > $ planetdiff planet-070307.osm.bz2 planet-070321.osm.bz2 > delta.xml
> > 
> > Applying a patch:
> > $ planetpatch planet-070307.osm.bz2 delta.xml > planet-070321a.osm
> > 
> > 
> > 
> > The tool is in SVN under utils/planetdiff/...
> > 
> > See the SVN readme.txt for further details or
> > http://trac.openstreetmap.org/browser/utils/planetdiff/readme.txt
> 
> Cool idea! But what is the reason not to use common GNU diff/patch?
> 

Try it and see what happens - diff tries to load both input files into
ram to process them. Fine if you've got a 64 bit machine with >6GB of
ram but not otherwise. 

I did wonder about changing the diff code to work differently but
figured it could not be too hard to write our own. It took a bit more
effort than I initially hoped but the result also provides benefits like
gzip/bzip2 handling and UTF8sanitize. 

It also opens the door for some analysis to be performed on the diff
files themselves which might (for example) provide some hints about what
tile need to be re-rendered.


Perhaps I should go back and try fixing up diff instead to compare the
results.

	Jon







More information about the dev mailing list