[Talk-ca] importing GeoBase Data (learning from TIGER)

Michel Gilbert michcasa at gmail.com
Thu Nov 27 00:30:02 GMT 2008


Hi all,

2008/11/26 Dave Hansen <dave at sr71.net>

> On Wed, 2008-11-26 at 01:40 -0800, Sam Vekemans wrote:
> >
> >
> >
> >         I do think it was important to have things broken up
> >         geographically.  It
> >         makes it much easier if something goes bad to find the data,
> >         remove it,
> >         an retry.
> >
> >
> > Since the data is broken up into Geobase tiles, perhaps importing by
> > tile area to get more specific. The provinces are rather large, so
> > going at it, by 1 degree x 2 degree would be better??
>
> Yes, it probably would be better.  However, there is also the problem of
> stitching things back together in the end.  I never dealt with that
> part.
>
> Also 1x2 degrees probably isn't bad for, say, the Yukon Territories.
> But, I would imagine that Toronto is going to fit almost entirely into
> one of those.  That might pose a few problems.  I think the largest .osm
> file that I uploaded was Kern county in California.  It was 165MB.




A regular tile size looks fine to me but will introduce duplicates along the
edges. How do we handle polygons that cross the bounding box ? From my
experience, uploading data with josm has a limitation number. Beyond that
limitation josm stops, therefore leaves a mess in the database. May be there
are others ways to upload large amount of data.




>
> >         One thing I never considered, but did come back to bite me a
> >         few times
> >         was concurrency.  I'd upload a node, make a way use it, then
> >         come back a
> >         few hours later to have another way use the node.  But,
> >         somebody got to
> >         the node before I did.  There were three or four of these and
> >         I fixed
> >         them up by hand.  It sucked.  :)
> >
> >
> > Well, remember (last week i think it was) when OpenStreetMap was shut
> > down for maintenance?
> > Well, what about convincing the foundation to shut down the server so
> > then all the data can be uploaded at once?
> > That would fix the problem that you had.  :)
>
> Sure, if you can pull this off, go for it.  Otherwise, it isn't *that*
> difficult of a thing to plan for and fix.
>
> Basically, if you notice that some node that you need is gone, you just
> re-upload a new copy of the original node and make a note of it.  It's
> that simple.
>
> >         Keep a record of everything that you do.  Keep good logs and
> >         make sure
> >         that whatever programs you use to upload the data can be
> >         stopped and
> >         restarted at any time with no ill effects.  This generally
> >         means keeping
> >         a table of which objects have been uploaded and their id
> >         mappings.
>
> >
> > I think we already discovered that the natural features shapefiles
> > data, shouldnt post any conflect... not a major one that is. ... Every
> > city does have some kind of water feature, and it's probably labeled,
> > but thats about it.
> > and for the other features .... ya pausing OpenStreetMap to make the
> > import happen.  would guarentee no point conflicts.
> >
> >
> >         I
> >         think bulk-upload.pl
> > http://wiki.openstreetmap.org/wiki/Bulk_import.pl
> >         does this pretty well, although I did have to
> >         customize it a bit.
> > Ya, as as far as i can see, the way that GeoBase keeps the data is a
> > bit different.
> > each province does have a different way of classing roads. .. so when
> > your literally traveling between provinces.. the pavement is
> > identical.. yet the signs on the roads indicate a different road
> > class.  (thats because provincial roads are funded provincially, there
> > is very little discussion between provinces.  So each provincial
> > upload would be different. (talk-ca talked about it back in the
> > summer)
>
> Yeah, one of the first steps is to come up with a conversion scheme to
> convert your features into OSM features.
>
> -- Dave
>

Geobase has a uniformed road classification. The matching between osm and
geobase road classes should be applicable globally. I suspect that local
contexts may be necessary in some cases (ramp classification form example).
We can start Geobase NRN - OSM Map Feature.  I can start this.


Cheers,

Michel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20081126/10040009/attachment.html>


More information about the Talk-ca mailing list