[Talk-ca] importing GeoBase Data (learning from TIGER)

[Dave Hansen]

On Wed, 2008-11-26 at 01:40 -0800, Sam Vekemans wrote:
>         
>         
>         
>         I do think it was important to have things broken up
>         geographically.  It
>         makes it much easier if something goes bad to find the data,
>         remove it,
>         an retry.
> 
> 
> Since the data is broken up into Geobase tiles, perhaps importing by
> tile area to get more specific. The provinces are rather large, so
> going at it, by 1 degree x 2 degree would be better??

Yes, it probably would be better.  However, there is also the problem of
stitching things back together in the end.  I never dealt with that
part.

Also 1x2 degrees probably isn't bad for, say, the Yukon Territories.
But, I would imagine that Toronto is going to fit almost entirely into
one of those.  That might pose a few problems.  I think the largest .osm
file that I uploaded was Kern county in California.  It was 165MB.

>         One thing I never considered, but did come back to bite me a
>         few times
>         was concurrency.  I'd upload a node, make a way use it, then
>         come back a
>         few hours later to have another way use the node.  But,
>         somebody got to
>         the node before I did.  There were three or four of these and
>         I fixed
>         them up by hand.  It sucked.  :)
> 
> 
> Well, remember (last week i think it was) when OpenStreetMap was shut
> down for maintenance?
> Well, what about convincing the foundation to shut down the server so
> then all the data can be uploaded at once?
> That would fix the problem that you had.  :)

Sure, if you can pull this off, go for it.  Otherwise, it isn't *that*
difficult of a thing to plan for and fix. 

Basically, if you notice that some node that you need is gone, you just
re-upload a new copy of the original node and make a note of it.  It's
that simple.

>         Keep a record of everything that you do.  Keep good logs and
>         make sure
>         that whatever programs you use to upload the data can be
>         stopped and
>         restarted at any time with no ill effects.  This generally
>         means keeping
>         a table of which objects have been uploaded and their id
>         mappings. 

> 
> I think we already discovered that the natural features shapefiles
> data, shouldnt post any conflect... not a major one that is. ... Every
> city does have some kind of water feature, and it's probably labeled,
> but thats about it. 
> and for the other features .... ya pausing OpenStreetMap to make the
> import happen.  would guarentee no point conflicts.
> 
> 
>         I
>         think bulk-upload.pl 
> http://wiki.openstreetmap.org/wiki/Bulk_import.pl 
>         does this pretty well, although I did have to
>         customize it a bit. 
> Ya, as as far as i can see, the way that GeoBase keeps the data is a
> bit different.
> each province does have a different way of classing roads. .. so when
> your literally traveling between provinces.. the pavement is
> identical.. yet the signs on the roads indicate a different road
> class.  (thats because provincial roads are funded provincially, there
> is very little discussion between provinces.  So each provincial
> upload would be different. (talk-ca talked about it back in the
> summer)

Yeah, one of the first steps is to come up with a conversion scheme to
convert your features into OSM features.

-- Dave