[Talk-us-massachusetts] Some observations and preliminary stats on MassGIS address import

[Greg Troxel]

Angela Morley <amorley at protonmail.com> writes:

> Interesting data.
>
> I've discovered an import process that works using QGIS, JOSM,
> OpenData plugin and Conflation plugin. I'll go into more detail in a
> little bit, but in involves using QGIS to modify the all capitals in
> the shapefile to partially lowercase (yes, this breaks things like
> McCabe Street, but I think that's acceptable. Community feedback,
> please?), and JOSM's Conflation plugin to assign the points to the
> closest building.

I don't think getting capitilzation wrong is ok; we can figure it out.

> There's a lot of little hiccups I'm discovering along the way, like
> how JOSM won't download town sized maps from the server, and you can't
> upload changesets greater than 10,000 edits. I'm not sure how to get
> around those hurdles at the moment.

I think the best path is to not use interactive tools like JOSM; the
building import was done with extracts, a database, and scripts.

>From geofabrik, you can get an extract of all data in Massachusetts:

  http://download.geofabrik.de/north-america/us/massachusetts.html

As for uploading, there are import tools that do chunked uploads and
handle errors well.  There's a lot that can go wrong, and Jason knows
most of them after doing the building import.

> Another facet -- should one person do the data conversion, and others
> upload the changesets? (that's fine with me, but it's going to take me
> some time going through 350 towns of data if I do this)

Remember that imports have to be with a dedicated account.

The building import was done by having a program that would produce the
changesets, and then those were published.  Many people checked them,
going over their town in great detail for hours, and then as we found
that there were essentially no hours, the level of review per town got
to be less and less.  I am pretty sure Jason actually ran the upload for
each town, once it was vetted by someone, but I don't remember.  In any
case, the work was almost entirely in getting a debugged preparation
script, and in reviewing the output.

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On July 17, 2018 12:57 PM, Yury Yatsynovich <yury.yatsynovich at gmail.com> wrote:
>
>> Thanks for creating the wiki page, Angela!
>>
>> I did some simple spatial joins of MassGIS address points with
>> existing buildings without addresses (either house number or street
>> name missing) using python (geopandas and osmnx). Here are some
>> stats from such joins by counties:

I just read about osmnx, and will check out geopandas.

It would also be good to know stats on addresses that are already in
OSM, what fraction are in massgis, and what fraction of those are within
some tolerance.