[OSM-dev] Representing differences between sets of OSM data

[Jon Burgess]

On Mon, 2007-08-20 at 14:04 +0200, Martijn van Oosterhout wrote:
> On 8/20/07, Brett Henderson <brett at bretth.com> wrote:
> > What are the reasons for attempting to use a common format?  Might sound
> > like a stoopid question but I'm curious what applications people have in
> > mind at this point.  At this point I'm not convinced that JOSM changes
> > should be represented in the same way as planetdiff/osmosis change sets
> > but I'm still undecided on the best approach.
> 
> Well, here is my use case (the reason why I brought all this up):
> 
> The AND data for NL needs to be merged with the current data. However,
> not everything we currently have is in the AND data and viceversa.
> Plus we also want to preserve some areas (ie. not remove them).
> 
> Now, I have script to cut out NL from the planet, and a script to
> determine what needs to be deleted (for reference we'll be deleting
> around 150,000 nodes, 180,000 segs and 30,000 ways). What I would like
> is that output for the last stage to be something I can apply to
> somewhere. The question is what format. We have:
> 
> JOSM format: currently only supported by JOSM, easy to generate. But
> the change file is nearly 50MB and I don't have enough memory to load
> this into JOSM.

How much do you have? I've just managed to successully load a 150MB osm
file by using 'using -Xmx2048m', but you wouldn't want to use it for any
interactive editing since it takes several seconds to respond to any
input but it works (this is on a machine with 4GB of ram). 

> planetdiff format: supported only by planetdiff. AFAICS it requires
> the diff to be "sorted" which my script won't do.

One useful tool to try is:
 svn.openstreetmap.org/applications/lib/libosm/test3

If you run "test3 file.osm" you'll find it should emit the file in a
sorted format (although you need enough ram for it to be able to load
the whole file at once).

> osmChange format: supported only by osmosis. It may be the most
> flexible of the lot, but it won't work for planetdiffs.
> 
> What I really want is two things:
> 1. From a given change file, apply it to a set of OSM data so I can
> see the results
> 2. From a given change file, be able to apply it to the server
> 
> JOSM gives me 2 (assuming I had enough memory). osmChange gives me 1.
> 
> The reason I'm asking is that it would be simpler if there was a
> format which could be well supported and be appropriate for most
> people. I don't particularly care which, and I wanted to know whether
> other people had any preferences.

There is another option I'm not sure you've considered: Run your own
copy of the rails server either with a complete planet data set or just
that of the Netherlands plus some surrounding area. 

You can then perform a synthetic upload either using JOSM or some other
upload script and verify the results (I don't know of any appropriate
upload script).

I could probably adapt the planetdiff to take in a JOSM-like file and
apply the actions but there would be no guarantee that the equivalent
action would work on the real API (e.g. the planetpatch tool has no
concept of referential integrity so will allow a node to be deleted even
if it is part of an existing segment).

	Jon