<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</head>
<body bgcolor="#ffffff" text="#000000">
I'm writing to get feedback and peer review on a new python module
to manage imports. And yes I am aware of the various concerns and
issues with imports, and the strong feelings they create.<br>
<br>
The goal of this module is to help make imports more robust and
correct over the long haul. As a side effect the module makes
imports dead easy to code.<br>
<br>
This came from a work-related project to mirror car sharing
locations into osm. The exact choice of tags and methods is subject
to review and I do seek your feedback. I wrote it a while back but
have dusted it off recently.<br>
<br>
<hr size="2" width="100%">The module performs a mirroring
operation. The external dataset must be a high quality and
authoritative source, with proper licensing.<br>
<br>
On the first run the import is conventional, except for some added
tags:<br>
<tt> source=osmfetch:xxx<br>
source:pkey=yyy<br>
source:licence=llll<br>
source:website=<a class="moz-txt-link-freetext"
href="http://zzz">http://zzz</a></tt><br>
<br>
On subsequent runs the module performs a merge-sort. Certain keys
are considered master in the source. For <a
href="iki.openstreetmap.org/wiki/Tag:amenity%3Dcar_sharing">car
sharing</a> that might be the count of cars at the location, the
phone number, and of course the primary key:<br>
<tt> contact:phone=510-555-1212<br>
</tt><tt> number=3<br>
</tt><tt> vehicles=Toyota Prius,Zastava Yugo,BatMobile</tt><tt><br>
operator=Hippie Car Cooperative<br>
</tt><tt> source:pkey=102</tt><br>
<br>
Other keys are left alone. Osm mappers are welcome to adjust the
coordinates for example (moving nodes more than 100 meters triggers
a warning to the car sharing operator, but the osm coordinates are
not touched).<br>
<br>
Destroying the <tt>source:pkey</tt> or<tt> source=osmfetch </tt>tag
disconnects the conflation process and could result in duplicates.
No record of the originally imported osmid is kept, and the entire
process is stateless.<br>
<br>
<br>
<hr size="2" width="100%">The import script is meant to be run on a
cron job, alerting a human when changes are ready to evaluate. The
output format is presently JOSM compatible XML, ready for human
review prior to merging. The JOSM file lists nodes to add, delete
and modify.<br>
<br>
This style of import is a perfect fit for the Car Sharing
application. I thought originally there would be many similar sets,
but in the end suitable sources seem few and far between.<br>
<br>
<br>
<hr size="2" width="100%">You can see some mirrored nodes here <br>
<a
href="http://taginfo.openstreetmap.org/search?q=osmfetch%3Accs#values">http://taginfo.openstreetmap.org/search?q=osmfetch%3Accs#values</a><br>
Prior to the import hand mappers had covered only 3 of the 150
locations.<br>
<br>
And what follows is an example control file. This one has never
been run live, as it duplicates an existing import. If run live it
would effectively update or freshen the import bringing in any
missing nodes and deleting obsolete ones.<br>
<br>
Thus an external dataset (be it corporate or community) can be
reflected without error or bit-rot in OpenStreetMap.<br>
<br>
What do you think of it? Is the python clear enough for general
use?<br>
<br>
<br>
<hr size="2" width="100%"><tt>!/usr/bin/python<br>
##<br>
## Author: Bryce Nesbitt, June 2011<br>
## Licence: Public Domain, no rights reserved<br>
##<br>
## DEMOSTRATION osmfetch module to import NOAA NEXRAD radar
stations<br>
##<br>
## See also:<br>
##
<a class="moz-txt-link-freetext"
href="http://wiki.openstreetmap.org/wiki/Potential_Datasources#Next_Generation_Radar_.28NEXRAD.29_Locations">http://wiki.openstreetmap.org/wiki/Potential_Datasources#Next_Generation_Radar_.28NEXRAD.29_Locations</a><br>
## <a class="moz-txt-link-freetext"
href="http://wiki.openstreetmap.org/wiki/Man_made">http://wiki.openstreetmap.org/wiki/Man_made</a><br>
##<br>
## Future work:<br>
##<br>
from osmfetch import osmfetch<br>
<br>
import sys, re, urllib, urllib2<br>
import zipfile<br>
<br>
from pprint import pprint<br>
from xml.etree import ElementTree<br>
<br>
class osmfetch_noaa_nexrad(osmfetch):<br>
<br>
# Sample noaa data:<br>
# <wsr><br>
# <name>KABR</name><br>
# <description><![CDATA[SITE:
KABR<BR>LOCATION:
ABERDEEN<BR>...]]></description><br>
# <Point><br>
#
<coordinates>-98.413,45.45600000000001,0</coordinates><br>
# </Point><br>
# </wsr><br>
def fetch_source(self, sourcedata):<br>
<br>
sourcenodes = {}<br>
plaintext_stream = zipfile.ZipFile(</tt><tt>osmfetch</tt><tt>,'r')<br>
tree =
ElementTree.parse(plaintext_stream.open('doc.kml'))<br>
<br>
for site in tree.iter('{<a class="moz-txt-link-freetext"
href="http://earth.google.com/kml/2.0">http://earth.google.com/kml/2.0</a>}wsr'):<br>
pkey = site.find("{<a
class="moz-txt-link-freetext"
href="http://earth.google.com/kml/2.0">http://earth.google.com/kml/2.0</a>}name").text.strip()<br>
description = site.find("{<a
class="moz-txt-link-freetext"
href="http://earth.google.com/kml/2.0">http://earth.google.com/kml/2.0</a>}description").text.strip()<br>
point = site.find("{<a
class="moz-txt-link-freetext"
href="http://earth.google.com/kml/2.0">http://earth.google.com/kml/2.0</a>}Point")<br>
lat,lon,ele =
point.find("{<a class="moz-txt-link-freetext"
href="http://earth.google.com/kml/2.0">http://earth.google.com/kml/2.0</a>}coordinates").text.split(',')<br>
<br>
node = {}<br>
node['tag'] = {}<br>
node['id'] = pkey<br>
node['lat'] = lat<br>
node['lon'] = lon<br>
#node['tag']['ele'] = (sites do not have
reliable elevation)<br>
node['tag']['source:pkey'] = pkey<br>
node['tag']['man_made'] = 'beacon'<br>
node['tag']['radar_transponder'] = 'NEXRAD'<br>
node['tag']['note'] = description<br>
#node['tag']['operator'] = 'NOAA'<br>
#node['tag']['website'] = <a
class="moz-txt-link-rfc2396E"
href="http://www.ncdc.noaa.gov/nexradinv/chooseday.jsp?id=">"http://www.ncdc.noaa.gov/nexradinv/chooseday.jsp?id="</a>+pkey<br>
sourcenodes[pkey] = node<br>
<br>
source_is_master_for=['operator','website','description']<br>
return(sourcenodes, source_is_master_for)<br>
<br>
# Map source primary keys to osm primary keys<br>
# Override only if your primary key is not stored in tag
'source:pkey'<br>
def map_primary_keys(self, sourcenodes, osmnodes):<br>
mapping = {}<br>
if osmnodes:<br>
for osmid,node in osmnodes.items():<br>
pkey = node['tag'].get('name')<br>
mapping[pkey]=osmid<br>
return mapping<br>
<br>
# Record the action (e.g. modify, add or delete)<br>
# Use your override to add, munge, or delete tags<br>
# If the action is 'none' but you want to change tags, alter
the action to read 'modify'<br>
def record_action(self, osmnode, action):<br>
osmnode['tag']['source'] = 'osmfetch:noaa:nexrad'<br>
osmnode['tag']['source:website'] = '<a
class="moz-txt-link-freetext"
href="http://www.ncdc.noaa.gov/oa/radar/nexrad.kmz">http://www.ncdc.noaa.gov/oa/radar/nexrad.kmz</a>'<br>
return(osmfetch.record_action(self, osmnode, action))<br>
<br>
#############################################################################################<br>
myfetch = osmfetch_noaa_nexrad()<br>
myfetch.run(description='NOAA NEXRAD Weather Radar Import',<br>
ext_url='<a class="moz-txt-link-freetext"
href="http://www.ncdc.noaa.gov/oa/radar/nexrad.kmz">http://www.ncdc.noaa.gov/oa/radar/nexrad.kmz</a>',<br>
osm_url=osmfetch.xapi_url +
urllib.quote('node[radar_transponder=NEXRAD]')<br>
)<br>
myfetch.output()<br>
</tt><br>
<br>
</body>
</html>