[Imports] [Talk-ca] Editing GeoBase data

Sun Oct 11 19:38:15 BST 2009

Hi James, talk-ca (and imports list, because this is an international issue
that is true for all imports) (and Joe for the LINZ data)

Great to hear from ya. :)

Although your question (see full message below) is in regards to GeoBase
NRN, im not the best person to answer that question (because i dont
personally do the actual conversion).

However, since i am now providing the data from the CanVec set, and (most
reciently) started providing the data for the GeoBaseNHN data.  Perhaps i
can shed some light.   ... um, ya i will anyway :-)

To get back to basics the purpose of osm is to "Mission: To map the world
and give the data away for free".

Taken, into context, we are building the maps at 1st by hand, and THEN as
data becomes available, we can use the available data to 'improve on the
existing map'.   (how i interpret that is to 1 - provide the data in .osm
format, then 2 - make this data available on a ftp server somewhere (ie.
mediafire), and 3 - claim the area your working on, and use whatever tools
that are best suited to bring this data into osm.

To explain further,
That said, I am of FIRM belief, that we need to treat 'importing government
data' (of all types), as an "addition-to" rather than "importing it all
half-hazardly, then fixing it, later". basically spending more time fixing
it, than the time it would have taken to go out and actually map it.

When mapping in general we are always concerned about: Map Quality & Map
Quanity.   However, having map Quantity isn't that great, when the quality
is poor.  and it's my FIRM belief that it's better to have empty spaces,
than having to go around and 'fix' everything afterwords.   We know that the
data source of 'accuracy in meters' (from geobase in particular) is a mix of
'acceptable' less than 10meters, to 'could be better' greater than
10meters.  And the age of the data, as well as the acquisition technique
needs to be taken into account.

So, as i mentioned above, for this 'data import process', we need to focus
on the step 1 & 2.   So when new data becomes available, we simply do step 1
and 2, and let the local area mappers view what is available, and they can
use what ever method they choose.  Here's a few examples of methods to use,
once the data is available on a server somewhere.

Note: - This data that is available (should) contain a copy of the origional
source files (ie. .SHP .GML .mp),  as well as a README.txt file (which
explains what the data is and how to use it), as well as a CHANGELOG.txt
file which lists what has changed since the last conversion-version.   And
of course the .osm file(s), which are appropriatly named. (and in my case,
the RULES.txt files that were used for the conversion) (and in others the
shp2osm.py script that was used, which contains the source tags 'to' osm
tags)

1 - Using JOSM, open up the .osm file (if shp-to-osm was used) it would be
less than 2000 points) and view it, and download as a new layer all the osm
area around the features, to ensure no duplication. and upload it.

2 - Using JOSM, open up the .osm files and download as a new layer the osm
area around the features, and merge the layers.  and saveAS a newname .osm
file.
2b - use some funkey script to find matches, then use some nifty way to
upload direct to the API
2c - using some funkey de-duplifier script would help if needed. :)

3 - using the source .SHP or .GML or .mp file (that is available within the
zip file of converted data), use OpenJUMP and some funkey magic to compare
it with existing OSM data and abra-cadabra & upload. ;-)

4 - using those .osm files upload to a tileserver somewhere out-there (like
osm.ca) and make a cool map.   That anyone can use. and add in select OSM
data as you please (thanks to ODbl) :) .. and maybe make it into a slippy
map that you can trace over too?

5 - using gpsbabel (i'll do that step for ya).  convert the canvecroads.osm
file into .gpx and open up the gpx file in JOSM and have fun tracing.
(especially handy for when your out there with walking papers, as you can
write in the streetnames, and when you get home you'll already have GPX
tracks that 'could' be better than what you have.. but it's a 2nd guide for
ya.)

6 - using OSMCUT with some funkey magic, you can take those .osm files and
slice them up even further (into 2, 4 or 16 or whatever pieces), and work
with what you like.

7 - and finally 7, (provided you are the ONLY one who drew in the map
features) and provided you have the DIRECT consent from local mappers (good
idea to make a meet-up-mapping-party for this),
   - load the .osm file in JOSM o(or using part of the above methods)
   - download a new layer of the current osm data.
  - delete the current osm data that you intend to replace (search select)
or manual select and remove.
       -hide the layer of to-be-imported.osm file
      - upload changes (removing map features)
      - show that to-be-imported.osm file and either merge with other, or
upload it

PLEASE  (but remember to note (on the wiki for small countries) or on the
GoogleDocs Chart for Canada) what tile your working on, so there is no
toe-stepping-on. And please if your going to load an area, please copy in
the WHOLE tile area, as that helps others when working on neighbouring
tiles.

So anyway James, as to try to answer your question, because we are building
our OWN map, and using the governent data as 'EXTRA', we are not depending
on this data, nor are we trying to make our map look like the government
data map. If there are areas where the government data can help, and using
on of the above methods will work just fine, then we will 'copy-in'.
(in-other-words) because the source .shp files are available in the zip
folder on a server somewhere,  The UUID, is UU-ISless (not needed).   When
new data is available it can be Again (step 1-converted & 2-shared), and
then compared to the older data.  and with some magic, compared with
existing .osm data.   Making us all :-) politely :)

And in summery, FYI, what i am doing is converting the canvec features and
the GeoBaseNHN features, and making these .osm files (and a copy of the
source) available on a server (for now it's mediafire.com) and will ALSO be
copying in the roads.osm files that those others are making.  and also
making these files available. (if anyone has areas they want done 1st, just
ask)

We did estimate 1 year for the process, and so far so good :)

By november, i should have a majority of the .osm files available from both
canvec & geobase NHN, so they will be abailable to put into the OSM database
taking most of next year. (with whatever method locals feel happy with).
(being sure what whoever is copying-in) makes contact (as much as possable,
to inform others of whats going on, so they can help too if they want).
(since i cant do everything) someone else can make the roads.osm files
available as that requres geobase2osm, super powers, that i dont have) (ie.
a basic PostGIS understanding)

Maybe (James) it doesn't answer your question, but should help the
imports at list understand more what's going on, as well as whoever is
following along
on the talk-ca list :)

Cheers,
Sam

P.S. Hopefully by the next conference call, i'll be able to explain it in
MUCH less words :) (so i can understand the positive & negative feedback on
this) :-)

On Sun, Oct 11, 2009 at 10:06 AM, James Ewen <ve6srv at gmail.com> wrote:

> We did a bulk import of GeoBase data provided by the Canadian
> government a while ago. I've been pondering and procrastinating on how
> to go about modifying the data without destroying it.
>
> The GeoBase import has a lot of information included with the ways.
> One piece of information is the UUID, a unique identifier for that
> particular way. We are told that the UUID needs to be kept with the
> way for future cross reference for future updates, as well as so that
> the Canadian government can find places where the OSM community fixes
> GeoBase problems.
>
> So, there are a number of issues that I am struggling with...
>
> 1) Modifying a way to add more detail/realign to reality.
>
> This one is easy, as you won't be destroying any data, only
> adding/moving nodes so that the way represents real world data better.
>
> 2) Deleting a way that does not exist anymore.
>
> Not too hard here either, if new construction has destroyed an old
> way, it needs to be removed.
>
> 3) Adding new ways.
>
> New construction, or areas missed by GeoBase need to be added.
>
> 4) Modification of existing ways that may destroy or highly modify data.
>
> This is where the quandary occurs. Things like chopping up a way to
> add bridges, or to realign an intersection, or other such
> modifications... do we just go about our editing with little regard
> for the existing data, or should we attempt to keep as much GeoBase
> data as possible? If we are to keep as much data as possible, what's
> the procedure?
>
> Maybe I have answered my question with 1, 2 and 3 above. Maybe I keep
> part of the existing way where it describes the way properly, or
> modifying it so it does, delete the rest of it, and then treat it all
> like new construction.
>
> James
> VE6SRV
>
> _______________________________________________
> Talk-ca mailing list
> Talk-ca at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/talk-ca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/imports/attachments/20091011/0d4cc878/attachment.html>