[Tagging] [Imports] New module to merge-sort imports over time (osmfetch python)

Bryce Nesbitt bryce2 at obviously.com
Fri Aug 26 01:31:57 BST 2011


On 08/25/2011 03:27 PM, Frederik Ramm wrote:
> Another problem which you don't seem to solve is the relationship of 
> imported and already existing data; it seems that in your particular 
> case you evaluated the situation by hand before importing. This is 
> often one of the more complicated things to do with imports, and it is 
> important to stress that even if you have your code right you still 
> have to do this analysis. We must not give people the impression that 
> importing was easy if only you have the right tool.
If the target data has a consistent key, the tool can prepare a 
preliminary diff.
For example I used initial search:
     node[radar_transponder=NEXRAD]
     node[operator=City CarShare]

Which brought in all the existing nodes to merge and compare. I was 
easily able to see what existing data needed preservation.

At the initial import time the conflation key can be changed:
     node[source=osmfetch:noaa:nexrad]
     node[source=osmfetch:ccs]

Unfortunately xapi is pretty weak at search, as you can only specify one 
key.  But one can override the osmfetch class to have a programmatic filter.
Some day maybe xapi will support rich search:

         *[radar_transponder=NEXRAD][operator=NOAA][country=us] |
          [man_made=beacon][type=NEXRAD]

And one can target a melange of hand mapped keys more readily.


> As you know, I am of the opinion that there should be no data in OSM 
> of which our mappers are not masters. If, in your specific case, a 
> mapper were to change the list of vehicles at a car sharing location, 
> I would consider it inacceptable to override this change from the 
> external source. If you want the external source to be the master then 
> mix it in at production stage, don't dump it on OSM.

I disagree here.  If the external source is "ground truth", then that's 
the data that should take precedence.  A car share operator for example 
won't want a disused location shown, and may well make that a 
requirement of permitting the merge-sort.  An import from a car share 
reservation system is definitive ground truth.

> And finally, we have a policy that says every import should be 
> discussed *before* it happens on a suitable mailing list or forum; 
> even if your 180-node car sharing station import may fly under the 
> radar, please do your bit to ensure that everyone who uses your 
> software follows the guidelines.
>
> My only technical comment regarding your script is that we usually put 
> the source tag(s) on the changeset, not the individual node, nowadays.

Ok, fair enough.

But I'd like to keep source tags on the node, so hand mappers can 
understand that the data is mastered  elsewhere.

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="osmfetch">
   ....
</osm>

Can you provide instructions for how to add changeset tags in the josm 
format above?



Also I'm worried about "source=osmfech:xxx" as a conflation key, as it 
may be more fragile than a custom key like "osmfech=xxx".



More information about the Tagging mailing list