[Talk-us-massachusetts] Waltham MA Data Import

[Greg Troxel]

Nicholas Davidowicz <nicholasdavidowicz at gmail.com> writes:

> I have been doing a lot of mapping around Waltham, MA. After reviewing
> other GIS providers (ArcGIS), I realized the city office does have very
> detailed GIS information available.

Welcome to the list!

> I am interested in attempting to import this data. My plan of action at the
> moment is first to contact the GIS office (the contact is Eric Rizzo),
> figure out the size of the dataset (or the relevant parts; my upper limit
> will be ~16TB, hopefully it should be well under this), and get the data
> transferred to my personal computer systems (either via internet or
> sneakernet to hard drives). Likely there will be a fee associated with
> acquiring the data, hopefully not more than several hundred dollars.

Before you start, I want to caution you that doing an import is very
difficult, and if you haven't been through one, it's likely 10x harder
than you think.  OSM is somewhere between anti-import and making really
sure the import goes well.

To do an import, you'll need to describe it really precisely, to the
point that people who don't want imports can't find anything to complain
about :-) This includes publishing the raw data as received, and the
code to transform that to the files that will be uploaded.  And an
explanation of translating shapfile fields (or whatever, but am guessing
.shp from the city) to OSM tags.

I would say the first thing is to get really clear on licensing.
MassGIS published data that is in the public domain.  I have no idea
whether state law says they have to, or it's just how they do it.  I am
not clear on individual town data, and whether that data is a work of
the town (by town employees), or if it comes from the state, or from
private companies, and how that affects the terms.  This is the biggest
issue that can make an import (or any OSM use) just impossible, and my
advice is to 100% solve this issue first.

My advice for the second thing (or in parallel) is to understand what
datasets are available and compare that to MassGIS, because it seems
there is a lot of data going back and forth town to state.

Another point is that if you can cause data to become available (on the
web, good licensing), you have done a great thing for OSM, even if you
don't get to import it yourself.  Every step helps and someone else can
pick up the work.

I don't want to discourage you from pursuing Waltham data, but another
thing on my eventual todo list (that I'm not getting to) is to look at
all the MassGIS data and see about importing more of that.  I would be
looking at 911 addressing and hydrography.

> Once I have the data, the target is of course physically survey-able
> information. Roads, sidewalks, building outlines, parking lots, fences,
> retaining walls, and water features are primary; secondary will be building
> addresses, trees, signage, cabinets, and other smaller features if
> available. I plan to import it one grid block at a time (the city is
> divided by evenly sized blocks numbered 1 to 81), starting with the city
> center 60 and working radially outwards. I will manually review all changes
> in JOSM and/or QGIS. This will likely take several months.

I can understand the desire to do manual review, but I think it's beyond
what's doable.  A few years ago, Jason Remillard led and did most of the
work to import the MassGIS buildings layer, and a bunch of us helped
review.   The idea was to script it all so that it would be safe, and
review enough to trust the scripts, and there was increasingly little
review as we got to the last towns.

A big point on an import is to not change data that's been hand mapped
(without hand verification, probably on-site).  So one pattern, used for
massgis buildings (my description is fuzzy), is to bring that data and
the osm data into postgis or something, and then to write some code to
iterate over the objects in the to-be-imported data and find out if they
conflict with existing map objects, and produce subsets of
nonconflicting/to-be-added, duplicates, and perhaps the rest.

Overall, this puts more time into automation and less into hand review,
so that the review is QC for the process, not checking each object.

The massgis buildings import was a huge success.   We have very good
buildings coverage, with no known damage to hand-mapped buildings.  The
only error was a bunch of things in the massgis database that aren't
really buildings (car tents, etc.), but that's really of no consequence
since they are sort of building-like.

I would be inclined to do it by feature, since I think the imports@
community will want uploads separated that way.

> If this goes successfully, and I still have interest, I may also approach
> neighboring towns in the future (Newton and Watertown first).
>
> Is anyone opposed to this? Does anyone have any tips or comments, or know
> if the city would even support this because of the licensing? I plan to
> contact Eric in a week or two once this post has had enough time for
> responses. Thank you for your attention.

I suggest that you print out the Contributor Terms and bring those.
Basically, if the city can say "PD", it will be ok, and if it's the
usual sort-of-free-but-not-really license, it will probably not work.

Another suggestion is that you prepare all the documentation that the
imports@ list will demuand before saying *anything* on talk-us@ or
imports at .  Several of us have been on imports@ for a while, and can
review/comment.  It is probably going to seem extreme, but it will go
much smoother if your proposal comes across as meeting requirements or
very close.

A useful contribution would be to list the possible data sources on the
OSM wiki and describe which of them could be added, how hard, what it
would improve, etc.  But probably digging in to the one you want is more
useful.

Greg (osm user gdt)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 162 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/talk-us-massachusetts/attachments/20170821/365b51bc/attachment.sig>