[Talk-ca] Data imports 3 part process

Sam Vekemans acrosscanadatrails at gmail.com
Mon Sep 28 19:51:33 BST 2009


Hi everyone,

I had a quick chat with Steve Singer yesterday, (just to finally talk over
the phone).   so i came up with this.

There is a need for really making this import (all imports) a 3 step
process.  The Data Conversion, the Data Sharing, and the Data Import.

1 - Data conversion, using whatever tool.  Be it, shp2osm.py or
shp-to-osm.jar or mp2osm (for New Zealand).   We make these files available.
  1a - What i recommend is in the .bat file
http://www.inthosedays.com/acrosscanadatrails/openstreetmap/canvec2osm%20conversion%20script/canvec2osm_v0_9_3/
(open it up, and save and view it as a txt file)
The 1st step is to copy all of the original source (in my case it's all the
.shp files for that tile area) so there is a back-up of it, and a recording
of the product version and all the details.   We do this, so that we have
something to compare with when newer versions are available
  1b - We go an convert all the files to .osm so that we have a copy of the
source files in the .osm format.   It's ALOT easier to play with the data
when it's in the .osm format, so we can re-convert if needed, so to add more
OSM tags, or change them.   (ie. change the 'attribution=*' tag where
needed, and the 'source=*' tag.
 1c - we have a copy of all the rules.txt file(s) so then those can be
changed where needed.

2 - data Sharing - By making these files available.  On a server somewhere
(for now im using my own, but not sure if my hosting provider is happy with
that).   So then these can be archived and saved, and stored in it's own
folders, so then new versions of the data can be available. And used to
compare.

3 - data import - by having all the converted files available on a server,
we can use a method like my GoogleDocs chart (or if it's a smaller import, a
WikiChart) and then people can 'claim' what tile ares that they are working
on, and be the 'main contact' for copying all the data over.  Sure, it's a
manual process.  However, what this means is that NO data is interfering in
anyway with what is already mapped by local area mappers.   The person who
is copying in the data is doing so VERY carefully, and only copying the
details that he/she feels are needed.  They will then update the wiki (or
googleDocs) chart with their progress, along with the date of last working
on it.

By going about it this way, we are treating the data that is available as a
ONE WAY import, so when newer data is available, we simply convert the data,
then share the data, and notify everyone that new .osm files are
available.
We can then use tools such as OpenJUMP 'AutoMATCH'  where we can pull the
latest data from OSM, and compare it to the latest .osm files that are
available and get a 'DIFF' file from it.  And go ahead and just copy in
those 'missing' pieces of data.

Cool tools- SHP-to-osm
What's wild about shp-to-osm is that it can be programmed to only look at
specifac features which are in the shp files, and using the rules.txt
program, and the -t option, we can 'filter' the data. to make the .osm file
we want

Hope all that makes sense,
Cheers,
Sam

Twitter: @Acrosscanada
Facebook: http://www.facebook.com/sam.vekemans
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20090928/9dd06fba/attachment.html>


More information about the Talk-ca mailing list