[Potlatch-dev] P2, snapshot-server, imports, vector layers and more

[Paul Norman]

> From: Andy Allan [mailto:gravitystorm at gmail.com]
> Subject: Re: [Potlatch-dev] P2, snapshot-server, imports, vector layers
> and more
> 
> On 15 November 2012 05:44, Paul Norman <penorman at mac.com> wrote:
> 
> > The
> > additional attribute prevents the files from working in JOSM
> 
> Oh, really? That sucks. I went for the additional attribute because I
> expect XML parsers to ignore it if they didn't understand it.

Osmosis, osmconvert and osmfilter all handle it, it appears to be just JOSM that validates it. Of course it'd be pretty easy to strip off the additional attribute, using any of those programs to read the XML then write it out with no changes would likely work.

> > but the bigger
> > issue is the use of positive IDs. When an object is merged in JOSM it
> > carries over its ID if it's a positive ID. This would lead to
> > conflicts with existing data in the API. Even if JOSM were patched to
> > allow it to work with these files and special-case their positive IDs,
> > using positive IDs seems like a recipe for disaster, particularly when
> a standardized format exists.
> 
> Well, I wouldn't say that negative ids are standardized, just that p2
> and JOSM use the same convention. I believe Merkaartor uses guids as
> placeholder ids, in contrast.

It's not the editors, it's the generators. Every converter from shapefiles to .osm that I'm aware of uses negative IDs by default. ogr2osm has hidden options to generate .osm files like osmosis expects (https://github.com/pnorman/ogr2osm/blob/8eb57cfcae743c3b348161f685e483e238288e1d/ogr2osm.py#L125) but shp-to-osm.jar and the old ogr2osm only handle negative IDs afaik. There are also CanVec, cat2osm and numerous other regional efforts that all use negative IDs. They have become the standard interchange format for converted importable data.

> Additionally, negative ids for background layers don't seem like the
> right solution to me, because that still gives a shared id space. What
> happens if you have three background layers, each with a node id="-1"?
> Or you create a new way in the main layer, and then pull through a way
> with a matching negative id?

As far as I can tell JOSM treats negative IDs as meaning these IDs are not assigned by the server so it assigns them to the ID space used for unuploaded objects. You can safely merge multiple files with negative IDs in JOSM.

A quick look with show info in JOSM seems to imply that there's a continually decrementing negative ID that persists for as long as JOSM is open, but I haven't read the code involved.

> Potlatch approaches this in a "more correct" fashion. Each vector layer
> has its own id space. When objects are copied to the main layer their
> ids aren't carried over - the main "osm" layer creates corresponding new
> entities, copies attributes, and takes care of the placeholder id
> assignment. I suspect JOSM will need to handle things in a similar
> manner. The "recipe for disaster" of using positive ids in vector layers
> is, as I see it, due to JOSM assuming that all the different layers are
> in the same id space.

As far as I can tell to properly work offline it has to assume positive IDs are in the same ID space or it would be unable to merge two .osm files that are both data from the API. Also, there is no main osm layer in JOSM. It's possible to have multiple files open that are all data from the API.

> So does snapshot-server serving negative ids actually solve the problem,
> or will it still lead to id space conflicts?

It solves the problem.

> 
> Cheers,
> Andy
> 
> P.S. I don't think there's actually any osm_id handling code in
> snapshot-server anyway, it just uses the same ids as are in the .osm
> file in the first place.

The problem comes from using osmosis to load the data. Osmosis requires positive IDs, versions and timestamps or the --read-xml task will error. Hence the shoot-self-in-foot options in ogr2osm so I could load it in. I'm not sure how I'm going to be able to load CanVec in - I may have to write some code that turns the negative IDs into positive ones, assigns versions and timestamps and figure out a way to merge it all together.

A postgresql query to invert all the IDs might work, but there'd still be the other attributes that aren't expected in an importable file.