[Talk-in] Mumbai

Shekhar Krishnan shekhar at crit.org.in
Tue Mar 10 16:24:39 GMT 2009

Pradeep, if it is a shp2osm tool you are working on, I look forward to 
hearing more. :)

See below for a description of OSM Extract-Transform-Load framework 
originally written by Schuyler Erle and Mikel Maron in Feb 2008.

Under the freemap project, we have resources to support anyone in India 
who wants to work on developing these tools for shp2osm and much more.

If anyone has links or suggestions for more temporary hacks, please let 
me know.


OpenStreetMap Extract-Transform-Load Tools

The current state of the art in the OSM community with regard to mass 
import of external datasets is extremely fragmented, typically demanding 
intensive custom scripting tailored to each individual data set. To 
facilitate the inclusion of further datasets, and to extend the state of 
the art in OSM mass import techniques, we would like to develop a 
generic Extract-Transform-Load framework for OSM (OSMETL) using the 
Python programming language.

The “OSMETL” framework will be developed approximately as follows:

Data Extraction: A class or set of classes for reading GIS data in a 
variety of formats will be created, using the Python OGR bindings from 
the GDAL project, a Free and Open library from OSGeo previously built 
for reading and writing standard GIS data formats.

Topology Creation: In the standard case where line strings will be read 
from a data set, a quadtree index of the line strings will be built, and 
then the intersections will be computed, to generate a topology from the 
original line geometries.

Attribute Mapping: A simple application-specific mapping language will 
be defined whereby combinations of attribute column and values in the 
original dataset are mapped to OSM key/value tags. The framework will 
employ a user-specified definition written in this language to generate 
a set of key/value tags for each node and line segment in the derived 
topology. If necessary, the framework will permit the use of a 
user-supplied Python module defining callbacks for computing tag values 
from a set of original attributes.

Reduction: For each set of lines and/or nodes sharing a common set of 
tags produced by the previous step, the framework will generate a 
description of an OSM “way” consisting of the constituent lines, nodes, 
and associated tags.

Export: Finally, the framework will output an OSM XML file, describing 
the ways and nodes generated by the previous steps. It will then be 
possible to load this output file into JOSM, resolve any conflicts with 
the existing OSM database, and then upload the result.

The framework will be designed in an object-oriented fashion, to make it 
possible to replace any of the steps of this process with custom code as 
needed. This work will form not only the basis of subsequent data 
imports performed on shape files, but will also be a very substantial 
contribution to the OpenStreetMap project at large, thus providing an 
opportunity to build links with the wider open mapping community.

Pradeep B V wrote:
> On Mon, Mar 9, 2009 at 1:35 AM, Shekhar Krishnan <shekhar at crit.org.in 
> <mailto:shekhar at crit.org.in>> wrote:
>     Like I've said on the list before, I'm waiting for someone to help with
>     converting these shapefiles into OSM.
> I have been working on this thing, will update once I am able to make 
> some progress.
> - P
> ------------------------------------------------------------------------
> _______________________________________________
> Talk-in mailing list
> Talk-in at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-in


Shekhar Krishnan
MIT E51-185
77, Massachusetts Avenue
Cambridge, MA 02139


More information about the Talk-in mailing list