[Talk-ca] What we dont know from GeoBase

[James Ewen]

On Thu, Jan 1, 2009 at 12:30 AM, Dale Atkin <datkin at ibycus.com> wrote:

> I'm sorry if some of the above has come off as a little abrupt, but I'm
> getting a little frustrated over here. I feel like I have a solution which
> should work, and should provide a better overall mapset for everyone, and
> provide means for updating with user data in a means that will be preserved
> across revisions (which I think was the main problem with wiping out the
> database and starting over with public datasources as a base), but I've not
> really gotten any feedback on it. No one has told me "well that won't work
> because 'x'. Or that isn't in line with the OSM philosophy because 'y'.

I read Sam's message last night, and thought about replying, but
declined. I probably would have written in a message that would have
been more abrupt. I raked Sam over the coals already, and didn't think
it would be good to do it again. Sam's trying very hard to make this
import a reality.

I appreciate Sam in the fact that he finds all kinds of stuff and
throws it out in front of me. Most of it I discard, but some of it is
good information that is of importance towards this project. We just
need to focus Sam on the goal, and take the distractions away!

Sam, forget about converting Ibycus to OSM. It's not the way to go. No
one besides yourself thinks it's the way to go. Even Dale says it's a
bad idea. Drop it, and concentrate on getting GeoBase data into OSM.
Forget CanVec as well... the project is GeoBase to OSM.

Dale, I seem to have missed out on your description of your solution.
Can you 'splain it to me again?

There are people working on scripts out there, but they seem to be
keeping them quiet. Maybe there's some communication happening on
other forums, but I don't see it here. I've been communicating with
two individuals who are making progress towards creating scripts to
start the upload of GeoBase data to OSM. Michel is one, and his
geopolitical boundaries upload efforts can be seen be all. There's
another individual working on an NRN import script. I'll leave it up
to him to decide when he wants to go public though. I don't have his
permission to disclose his information.

> Instead we just seem to be talking in circles getting nowhere.

I don't think so, Dale... You're just getting dizzy watching Sam run
in circles! 8) The rest of those with comments all seem to be headed
in the same direction.

I see only 2 real hurdles to be overcome:

1) Decide on which GeoBase attributes get mapped to which OSM attributes.
2) Determine how to merge the new GeoBase data into the OSM database
while still adhering to the OSM Code of Conduct.

We're currently making some progress on #1, with the wiki page showing
proposed attribute mapping.

I have a proposed concept of how to allow the GeoBase data to be
merged into the OSM database. It's kind of an offshoot of Sam's
concept, but bear with me.

In Potlatch currently, you can click on the little "+" sign on the
right hand side of the display, where you can choose one of 4 base
layers, and one of 2 overlays. If you choose the data overlay, you get
an object list of all the ways in the displayed area. From there, you
can click on the way number in the object list, or click on the way in
the map view. Either of these presents you with the attributes
associated with the selected way.

My suggestion is to add a third overlay capability, that being a
GeoBase layer. All of the GeoBase import would be created at once, but
held off of the main map in a pending database. To be incorporated
into the OSM database, one would simply need to go to the area of
interest, and bring up the GeoBase overlay. If there were no visible
conflicts on screen, a click on the "Import All" button would merge
the visible GeoBase data into the OSM database. (More on this later,
making zero conflict areas automatically imported)

In areas where existing OSM data conflicts with GeoBase data, the user
would be able to select the existing OSM way to check for any
desirable tags, and have them added to the GeoBase data. The OSM way
could be deleted and GeoBase way inserted into the OSM database one
way at a time. (There are issues like ways that extend beyond the
extents of the visible area.)

I am favouring adding OSM tags to GeoBase ways to get the GeoBase
nodes in place for future comparison. I am guessing that for future
GeoBase updates, we would not only look at GeoBase ID numbers, but
also at node locations. Some relocation of nodes by OSM users will
happen, and having the ability to find these discrepancies would be
nice. A node relocation might have happened by accident, or by design.
Having the future updates highlight this would make it easier to know
if the update should happen, or be left behind because the OSM data is
newer or more accurate.

For future updates from the GeoBase database, the script would compare
GeoBase IDs, and node locations to note whether there has been any
changes made to the data. Where changes are noted, the GeoBase data
would be included in the GeoBase layer. Areas where data had been
wiped out, or not yet incorporated would show up. Areas where the new
Geobase information matches the OSM data, would be left out. This
would end up looking something like the Maplint layer, where only
errors show up.

If the initial import script is written cleverly enough, it could work
on a tile by tile basis, looking for any OSM data. If there's no OSM
data to conflict with, automatically import the GeoBase data. If it
finds OSM data, then subdivide the tile down and check again uploading
areas of no conflict. Obviously there would be a minimum tile size
set. For those tiles that have existing OSM data, create a GeoBase
data overlay layer for that tile, and ask local OSM users to vet the
data for that area. Conflict areas would be easy to spot, as the
GeoBase information would be missing from the map. Areas of
non-conflict would end up with all the GeoBase data showing in them,
and areas of conflict would have only sparse OSM data displayed.
(Thinking of areas where only a single highway runs through the
coutryside. Highly developed areas like downtown Toronto would
probably not be that easy to spot, but a look at the GeoBase overlay
would show where work needs to be done.)

The big concern is not tossing all of the hard work of the existing
OSM community out the window, and starting from scratch with a bulk
GeoBase import. By doing a smart import, importing only into areas of
zero data, we can get the automation necessary to fill a lot of the
uncharted territory, but still respect the work of all those who have
been contributing up till the import.

Manual intervention at some point is something I don't think we can
avoid. We need to put RI (real intelligence) into the project where
our AI fails.

This concept bridges the gap between Sam's pylons and a wholesale
slaughter of OSM data in favour of GeoBase data.

What has to happen to make this concept work is to get some
programming added to the Potlatch, and/or to the JOSM tools. We need
to have the ability to see both the proposed GeoBase data, and the
existing OSM data, to allow the human intelligence to make the best
combination of the data present.

Even without the support of the editors, we could make the scripts do
the zero conflict tiles import right now.

What say you?

James
VE6SRV