[Talk-ca] importing GeoBase Data (learning from TIGER)

[Sam Vekemans]

>
>
>
> I do think it was important to have things broken up geographically.  It
> makes it much easier if something goes bad to find the data, remove it,
> an retry.


Since the data is broken up into Geobase tiles, perhaps importing by tile
area to get more specific. The provinces are rather large, so going at it,
by 1 degree x 2 degree would be better??


>
>
> One thing I never considered, but did come back to bite me a few times
> was concurrency.  I'd upload a node, make a way use it, then come back a
> few hours later to have another way use the node.  But, somebody got to
> the node before I did.  There were three or four of these and I fixed
> them up by hand.  It sucked.  :)
>

Well, remember (last week i think it was) when OpenStreetMap was shut down
for maintenance?
Well, what about convincing the foundation to shut down the server so then
all the data can be uploaded at once?
That would fix the problem that you had.  :)

>
> Keep a record of everything that you do.  Keep good logs and make sure
> that whatever programs you use to upload the data can be stopped and
> restarted at any time with no ill effects.  This generally means keeping
> a table of which objects have been uploaded and their id mappings.


I think we already discovered that the natural features shapefiles data,
shouldnt post any conflect... not a major one that is. ... Every city does
have some kind of water feature, and it's probably labeled, but thats about
it.
and for the other features .... ya pausing OpenStreetMap to make the import
happen.  would guarentee no point conflicts.

I
> think bulk-upload.pl

http://wiki.openstreetmap.org/wiki/Bulk_import.pl

> does this pretty well, although I did have to
> customize it a bit.

Ya, as as far as i can see, the way that GeoBase keeps the data is a bit
different.
each province does have a different way of classing roads. .. so when your
literally traveling between provinces.. the pavement is identical.. yet the
signs on the roads indicate a different road class.  (thats because
provincial roads are funded provincially, there is very little discussion
between provinces.  So each provincial upload would be different. (talk-ca
talked about it back in the summer)


> Try to contact the 'owners' of local areas.  Most of them will be a bit
> cranky,

We haven't heard from some people yet. .. and many of the smaller
contributors don't know it yet.  I did post it on the Diary board.. so more
people will get the word.
I think the number is about 20 larger users? I'm sure we can get an exact
number on that.

but will grudgingly accept that they generally need to clear out
> their work and just deal with the new data.  But, let them make the
> decisions as much as possible.
>

So a bounding box over the 'complete' areas... or a blanket 'render=no' over
the whole thing. So it shows 'ghost lines' where the difference is from the
import to what the users did.

>
> I'm sure you're also going to do plenty of small-scale experiments.


I think taking some screenshots of what the ibycus topo looks like on
mapsource, vs. what the OSM data looks like on mapsource, is a good example
to show people the differences, would really help.


>  I
> had to do several iterations, mostly on my local data set before it
> starting rendering like I needed.
>
> > My thought was adding the render =no tag to it all. and import it all,
> > then manually going in there and cleaning up the roads which don't
> > align the same way.
>
> Oh, you mean for overlapping data?  There's really no good way to handle
> the overlapping stuff.  I'd say just import it all, keep it visible,

Well, we can show screenshots of what the Mapsource view of both the Ibycus
topo, as well as current OSM data, will look like, with all its ghost lines
showing.?

> and
> let people clean it up later.
>
> -- Dave
>
> By having the data available, in some cases where the town only has 2 lines
running through it, it would be easier to blanket import the data, and
select it all and change it to render=yes. .. and then eithor move the
imported data points so it overlaps the user data.. or move the user data so
it overlaps the imported data.
By having everything set to render=no, this avoids rendering the ghost
lines, and there is no cleanup needed for the 'completed' areas. .. so this
can be done in potlatch easily, as it will show these not rendered roads, as
ghost roads. (They show up lighter on potlatch anyway)
.. so really... if someone is really board.. they can go in there and move
the user data ontop of the imported data if they like... but it would really
make no difference. ... as long as the line that is rendered is right. ..
ya, no difference.

.. and if your working on an area. .. the 1st thing we would ask users todo
is select the areas and get it all renderable.  in JOSM. .. with potlatch,
its visually easier to see whats not rendered, but for a new area.
...
So this way... for when i mapped protection island.... none of my work would
be a waste of time. .. i would simply be able to view the imported data, and
quickly see where there is more that can be added... and I can also see
where it wasn't available.

So road classification is just 1 step... the next is surface type. .. on
protection island.. it's all gravel roads. .. still residential, but it
needs to be noted.


So thats some ideas:
-Close Openstreetmap database for maintenance, then upload all the data at
once.  (after we figured out all the details ...of course)
-having it all as render=no... which would keep everyone happy.
-is it possible to have a plugin which would make it possible to show/hide
certain types of data. .. so the geobase data could be hidden? is this
needed?
-doing major test areas (to make sure the classification and point and area
types are all as accurate as possible)
... which they should be, if were using the same system for the Ibycus topo
get them all right.  .. as the basics is the same way to make it show up on
Garmin GPS how it 'should be'.

Of course, there will be lost of work todo after the import, and the chart
of all the known in consistencies would be helpful... or maybe not
needed?... as any experienced user would see it right away. .. and go ahead
and change it.

Hopefully this all makes sense, :)

Cheers,
Sam Vekemans
Across Canada Trails
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20081126/b3ad6b46/attachment.html>