[Talk-ca] Sam's summery essay (was Re: Correcting Geobase_import_2009)

[Sam Vekemans]

Hi John,

Sorry for the delay... and thanks everyone else for chiming in.  Others can
(as you saw) explain it better. :-)

This message is directed to the talk-ca list, as it serves as a summery for
the latest and greatest.   In a month or so, I'll be able to summarize in 1
page.   But for now, I've put a lot of thought into the below message, so,
although long and rambley... it's the best answer i got :)


First off, let me introduce the team... lets call it 'the GeoBase Road
Crew'.  These folks all know how to use python scripting and know how to use
a 'postGIG' database.  And are brilliant!  And going a super awesome job!
>From west to east, we have;

Steve Singer - who loaded 092c area for me - And kicked of the whole
province of Alberta & Most of Ontario... doing a super job .. and we have to
thank for all the hard work done. .. paving the way for the others to
follow.

Jason Reid - who started the geobase2osm script (i think getting the initial
geobase tag matching done) revising the script from shp2osm

user: Austin Henry - Who has converted All of Vancouver Island.   And for
the 092B (Victoria area) he made the complete.osm area file available.   As
it was easier to manually add in roads.. than to pick-and-choose from
RoadMatcher.   (Since there's a lot of mappers around, this makes sense).
And for the rest of the island, he manually checked out roadmatchers results
and tried to attach as many geobase roads to OSM roads as he could.

user: Michael Barbanov - Who is working on the 092G area (Vancouver area)
where 092G07 (port Coquitlam) is awesome.

User: Neskie - who is in the 082L area and did some roads first, then went
on to be in charge of the Aboriginal Lands Database..

user: Adam Dunn - who is now going full at it, and getting (what looks like)
the rest of BC done.  Starting with 092H Wow, awesome!

user: Steve Singer - again did all of Alberta, and most of Ontario

user: John Peterson - who did some of southwestern Ontario (and used the
Stats Can roadnames) before roadnames became available in Ontario, using
that cool script to match it up.  Awesome!

user: Frank Steggink - who is doing a super job in Quebec - getting it all
done.

user: Yan Morin  - who did the Mont Laurier area .  and also started the
geobase NHN data, and make a cool python version of the script.

.... and of course, the entire talk-ca list, as anyone who has ideas, chimes
in when they can :)

... and me user:Acrosscanadatrails, who doesn't know a think about python
except that it works great, and we all like the results. :)   I'm the one
who's keeping track of the progress on a national level using the 'Canada
Data Import Chart' [1].
... I'm also the one who is managing the conversion process for the 'CanVec
data (all the other map features).  For this conversion I'm using Ian Dees's
script shp-to-osm (which is simple java script and command line function
with a rules.txt file).  Who's helping along.
http://wiki.openstreetmap.org/wiki/Canvec2osm

.. as well, I'm working he geobaseNHN data (that user: Yan Morn started) and
found errors).  Frank is helping right now get the details on
that.(considering that the data can also be found on canvec.  So I'm sorting
all that out.


We now have a imports@ discussion list[2], where the folks around the world
who have conducted (or will be) or interested in.  dealing with large
amounts of data, and how it effects the community. etc.   Brilliant minds,
where I'm amazed everyday.  How ALL questions can be answered.

And so, (recently) I've been using the IRC #osm channel[3], where there are
on average 150 awesome mappers/programmers/GIS professionals/architects...
and more who basically can answer any question asked. ... and debate about
that, with fun and sometimes heated discussions. ... always ending up in
some kind of change for the better :-)

Also, the talk-au & the talk-us list, where in the USA, there dealing with
the post-Tiger trauma... and in Australia where they now have access to all
this data and working it all out.    And Also, new Zealand, where they are
working on converting .mp files converting to OSM (LINZ data).[4]

On Sun, Oct 25, 2009 at 9:43 AM, john whelan <jwhelan0112 at gmail.com> wrote:

> Yes I'm new and my background is red tape.  I understand there is a
> lot going on and I appreciate the work that has been done.
>
> My background is in databases etc. If you ask clients what they want
> they always rate reliability above anything else.  To me the data from
> geobase is good high quality data with input from multiple sources
> including municipal governments.  Talk to them nicely and we can get
> all sorts of high quality things like bus stops, speed limits etc
> direct from the municipalities.
>


So ya, right now i have a stack of data available,, and ready..  I DO think
that the community would appreciate me just doing one thing at a time.   I
volunteer to get it done. Others can help if they like.   But the community
would agree that getting all the roads in is 1st priority, then all of
CanVec data in is next... then the national protected areas... then the
aboriginal lands... then any Land Information Ontario that got missed from
CanVec. ... then the Nanaimo Data thats available.. then the Vancouver bus
stops that are available.
As thats a logical order :)   (and geobaseNHNrivers are mixed in there with
canvec, but some geobaseNHN stuff too, depending on the back-end details for
that)

.. and once municipalities get wind of whats going on, they'll probably pop
up :-)  and the CRD (Capital regional district, is on it's way, but probably
a few years away... might be because they feel left out since Vancouver
opened up there data :-)


> What appears to have happened is where the data has been merged roads
> that are in the geobase database no longer connect to roads that have
> been put in via potlatch.  I think the end point is dropped

  The older potlatch roads do not have the same depth of tags as the geobase

> data.  Not only that but roads that run close to the old roads appear
> to be deleted for the section close to the potlatch inputted roads.
>

Here's the low-down. (social impact)
We respect the integrity of the local area mapper who spent a considerable
amount of time either tracing from imagery, or tracing from there own GPS
tracks....  and place this on a HIGHER priority than that of geobase/canvec
data.
So, again... this is Openstreetmap, where its a collaborative community who
builds the map. ... we respect the integrity of the local area mapper who
spent a considerable amount of time either tracing from imagery, or tracing
from there own GPS tracks....  and place this on a HIGHER priority than that
of geobase/canvec data.



> Oh and my WAAS enabled GPS thinks at least one Potlatch inputted road
> is in the wrong place.
>

.. the more GPS tracks along the road the more probability of accuracy :-)
 .. and i use garmin too ...that device is technically the best IMO :)
http://wiki.openstreetmap.org/wiki/Garmin


>
My personal view is that we should go with the geobase database and
> see if we can layer on tags etc.  That way we can update the database
> from time to time by replacing the entire geobase data layer.
>
>
So by having this .osm file available in JOSM as under the current data.osm
layer.  You can manually copying moving the ways over, and join what you
like. (selecting a whole residential suburb, if it was missed)

If there are no other mappers around, YOU are now the 'expert' of this
area,  you can make it as detailed as you like.


> I think OSM can add value in Canada basically by tagging and enhancing
> data already there, if a client can get geobase data elsewhere that
> actually has connecting roads and complete roads they may prefer that
> to OSM where the data quality isn't so consistent.
>
>
We don't provide data for 'clients', rather, were shellfish in that sense (i
live by the ocean, its shellfish),  as we make the map better for
ourselves.  ... and as a result, YES our map is superior to other maps of
similar quality.  (but. of course,  inferior to engineering drawings)

For all the other map features... here's the low-down   (unlike, the US
TIGER data, this is different).  I learned from Dave Hansen who did the
initial Tiger Import, and he shared the experiences, and ideas.  ... as well
as from when Michal Gilbert (who did the province borders)  as well as from
Steve Singer (who did the initial Alberta import, and Ontario import).

As a community, we decided (back in December 08) that we need to do what we
can to preserve the integrity of the local area mappers who spent the
previous years driving the thousands of km all over the country.  and local
cities of course too, where lots of mapping has been done.
... so the end result and the SOLUTION is that we make these .osm files
available and local people can do what they like. ... they can have the
geobase_road.osm file available (download the file and open it) and move the
osm roads over a bit to better align where needed.

AND it's those folks who i mentioned in the beginning of this letter, that
are working hard and will get it all done.   And so, for those areas that
are slightly more complicated, they will make the complete.osm file
available for local people (like yourself) to manually copy in what you
like, and manually moving the existing data slightly (but only if your
adding more features the the map).

So anyway,
I make these .osm files available and they will be stored on the NRCan ftp
site (thanks NRCan :-) ... basically, a direct copy-translation of the
canvec SHP files, and converted to the OSM equivalent.   The files are made
available in bit-sized files (so the API can handle 2000 points max at a
time).   This allows you to open up your local area (NTS tile) and open up 1
file at a time, and pan around the area and load the osm data wherever the
features area.

BTW, if possible, i will ALSO make the canvec roads. (This is a direct copy
of GeoBase roads) available, along with the .zip package.   So then they can
be used as a reference.   (They are available in the next version of
canvec-to-osm, as i run the script, but with the duplicate nodes as a KNOWN
ERROR) (i labeled the file to indicate that) :-)

 ... ie loading the 'railways.osm' file you would download all the data
along the railways.... and find out what has/hasn't been copied in.  (you
can ALSO open up a file that is the 'NTS boundary area rectangle'.osm  so
you can use that as a guide. (but don't upload the rectangle, as its not
needed)

... so,
You then need to find out WHO it was who put those railways in there that
were there already, and contact them.  (as they might want to help you, and
as a courtesy to the local other mappers ... as thats how we map :-).  .. so
if the railway is off by more than 10 meters, its fine to adjust them.  if
MORE than 10 meters.  then you'll need to keep that original in there until
someone can manually go out there and verify.  (alot of times the data
available might be wrong or old).

here's an example... take a look at the contours. .. it's kinda hard to tell
whats better.. as there might be a 'spur' that goes off..  i don't know for
sure..
http://www.openstreetmap.org/?lat=48.5124&lon=-123.5481&zoom=14&layers=00B0FTFT

    Just because it came from CanVec government data... doesn't always mean
the accuracy is there. (ortho-rectifying) from paper maps at a different
zoom, may make a BIG difference.   ... and this railway data also included
the bridges, which might have not been mapped by others.  (so an easy way is
to break the way at either end of the bridge.. and add the tag 'bridge=yes'
to it.

...and another option...   lets say you want to run bulk_import and OpenJUMP
. .. well, i make a .zip file available in this same .zip package that
includes the original shp.zip fileset that came from the NRCan site.   so
you can then load the local .osm area using mySQLpostGISthing (or
whatever)... and get a DIFF file. .. then convert that to osm.  a derivative
of geobase2osm can do the trick.. or if it's a SHP file, the basic
shp-to-osm would work.   (if the area your working on has no data to begin
with in the 1st place, then direct API load is fine). ... as its IDENTICAL
to the way JOSM works... accept using the command line for it.

... IMO  (from experience)  Bulk_uploading data is a 'blind' way of
importing data. So to just convert DIR|ECT from the SHP file to OSM and to
the API.... without creating .osm files 1st, and actually looking at them to
verify what it actually is that you are loading....   this is why i STRONGLY
recommend taking the slow approach, and opening up 1 file at a time...

BTW, to load a full NTS area of your working area.... the zoom is set that
it takes just 6 spots panning around the rectangle to grab the whole NTS
tile area.  (not much).
... and hey, if your file is to 'spotty' where there are bits all over the
place, you can just hide the current OSM area, and manually just select the
data you don't want to load right now, and delete it, and then re open up
the file and select the other half.   (if that makes sense)

... So anyway, the point is this.   since YOU are the one who is copying in
the data to OSM, you have the responsibility to ensure that the quality of
the data you are dropping in.. is the best that it can be.  ... and if it's
NOT... well, you shouldn't be coping it in.

see specifically
http://wiki.openstreetmap.org/wiki/Import/Guidelines#Create_a_community

... as 'plopping-in' data just because it's available is NOT that great.
(for Canada, OpenJUMP with Automatch is the best solution.  .. however,
having the roads as NTS tiles and dropped in locally might have been a
little better.... but overall, since the size of the country is so great..
automatch did a super job, and didn't interfere with the local data, which
is what the goal was.



> Oh and I prefer JOSM to potlatch for some reason.
>
> For loading .osm files... JOSM is the tool to use.  I use potlatch when
adding relations, because its a bit easier.  .. and tracing off of yahoo
imagery is easier also with potlatch... and simple small edits too.


Thoughts?
>
>
Yup... thats good for now.

Cheers,
Sam


> Cheerio John
>
> 2009/10/25 Sam Vekemans <acrosscanadatrails at gmail.com>:
> > Hi,
> > just use potlatch & connect the roads. No harm done :-)
> >
> > The OSM changeset history keeps track of attribution and such.
> >
> > If you want a copy of where the geobase version of that road is, i can
> > provide that.
> >
> > The system that imported the roads didnt want to interfeer with what
> > work previous people did.
> >
> > Cheers,
> > Sam
> >
> >
> > On 10/25/09, john whelan <jwhelan0112 at gmail.com> wrote:
> >> Looking at my local map in Orleans Ontario noticed that Merkley Drive
> >> does not connect up to Charlemagne.  Merkley Drive is tagged
> >> Geobase_import_2009 and all sorts of interesting things.  Charlemagne
> >> is just tagged potlach and residential road.
> >>
> >> Is there an easy way to extend Merkley Drive so it does connect?  ie
> >> add another node but copy the attributes of the existing road?
> >>
> >> Should I simply hold off until I know that the Geobase_import_2009 has
> >> been completed?
> >>
> >> Is there a way to submit a GPS trace into Geobase to get their base
> >> map corrected?
> >>
> >> Thanks John
> >>
> >> _______________________________________________
> >> Talk-ca mailing list
> >> Talk-ca at openstreetmap.org
> >> http://lists.openstreetmap.org/listinfo/talk-ca
> >>
> >
> >
> > --
> > Twitter: @Acrosscanada
> > Blog:  http://Acrosscanadatrails.blogspot.com
> > Facebook: http://www.facebook.com/sam.vekemans
> > OpenStreetMap IRC: http://irc.openstreetmap.org
> > @Acrosscanadatrails
> >
>

[1]
http://spreadsheets.google.com/ccc?key=0Am70fsptsPF2dG1ZN1YwMmZCVDhDOHZpbUNmOGlvWGc&hl=en

[2] Import at openstreetmap.org - discussion list

[3] http://irc.openstreetmap.org/irc.cgi

[4] talk-us at openstreetmap.org; talk-au at openstreetmap.org;
http://wiki.openstreetmap.org/wiki/LINZ
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20091028/59f8eafc/attachment.html>