[Imports] [Talk-us] Import WestCOG building footprints in south-west Connecticut

[TC Haddad]

We were fairly low tech on tracking tasks. I don't know that taskscheduler
existed at the time.

Instead we posted a list of the data tile names on github, and then each
person who was contributing marked the file they were working on, and
marked it again once complete. That page was here:
https://github.com/pdxosgeo/pdxbldgimport/blob/master/data_to_import/Import_Tasks.md

On Sat, Sep 12, 2020 at 7:51 PM Martin Machyna <machyna at gmail.com> wrote:

> Thanks for catching the 'null' values. In geopandas data.frame each
> row/object must have assigned values for all fields. I was just assuming
> null values don't get exported.
> I guess this can be fixed with something like: sed 's/, "[^"]*": null//g'
>
> How did you keep track of your import progress? Did you use taskscheduler?
> We are looking for something that, but without public access.
>
> On Fri, Sep 11, 2020 at 8:04 AM TC Haddad <tchaddad at gmail.com> wrote:
>
>>
>> Hi Martin, I have limited experience with imports, but I helped in
>> Portland, OR. I looked at your Westport file Two things I would pass on:
>>
>> - if the field is null, I don't think the tag should be in the import.
>> Null fields make the files unnecessarily large and provide no value
>> - we removed the buildings that conflicted with existing buildings in
>> advance
>> - we had a separate process to handle buildings with multiple addresses
>> - we tiled the geography into squares of no more than 500 buildings each.
>> This makes it possible for an import volunteer to have a workflow where
>> they import a square and then review every building for the conflicts you
>> mentioned (roads crossing buildings, etc). More buildings imported at one
>> time makes the review very difficult, especially if there are lots of such
>> corrections
>>
>> Our Github was here in case there is anything else of value to you
>> https://github.com/pdxosgeo/pdxbldgimport, and our wiki page was here:
>> https://wiki.openstreetmap.org/wiki/Portland,_OR_Bldg_import
>>
>> On Thu, Sep 10, 2020 at 6:31 PM Martin Machyna <machyna at gmail.com> wrote:
>>
>>> So to update this thread, I have integrated addresses from CT Open Data
>>> dataset and also updated the wiki page (
>>> https://wiki.openstreetmap.org/wiki/Connecticut/Western_COG_Building_Import
>>> )
>>>
>>> The whole dataset can be looked at:
>>> https://drive.google.com/file/d/10hUl09WSmK-I8h1hkMlELIGwDYsEziSe/view?usp=sharing
>>> For quick loading I also made a subset for one town (Westport):
>>> https://drive.google.com/file/d/1oMSbHXpPY5eLSHhGlikGy2uudyq3VyoV/view?usp=sharing
>>>
>>> If there is any issue, please let us know.
>>>
>>> On the side note of the CT Open Data buildings suitability for import I
>>> found a way how to simplify all buildings in an automated way in python.
>>> Here is a quick comparison of before/after
>>> https://files.slack.com/files-pri/T029HV94T-F01AD0FNFD3/simplified.png
>>> (you need to have Slack account)
>>> It doesn't look so bad and we could consider it for a next round of
>>> import.
>>>
>>> Just for a future reference in case someone would need to do the same,
>>> the python code is:
>>>
>>> import geopandas as gpd
>>> import pandas as pd
>>> from shapely import speedups
>>> speedups.enable()
>>>
>>> address =
>>> gpd.read_file("Connecticut_Buildings_with_Addresses_experimental.shp")
>>> simple = address.simplify(0.000005, preserve_topology=True)
>>> simple.to_file('Buildings-simplified.geojson', driver='GeoJSON')
>>>
>>>
>>>
>>> On Sat, Aug 29, 2020 at 6:50 PM <joe.sapletal at charter.net> wrote:
>>>
>>>> I was going to look at the buildings too.  I’ve used a tool in ArcGIS
>>>> to correct some pretty awful buildings, but I couldn’t download them
>>>> either.    If there is no hurry, I’d check in again with the contact on
>>>> Monday.  It would be nice to have the buildings with addresses on them.
>>>>
>>>>
>>>>
>>>> Joe
>>>>
>>>>
>>>>
>>>> *From:* Yury Yatsynovich <yury.yatsynovich at gmail.com>
>>>> *Sent:* Saturday, August 29, 2020 5:05 PM
>>>> *To:* Julien Lepiller <osm at lepiller.eu>
>>>> *Cc:* imports at openstreetmap.org; talk-us at openstreetmap.org
>>>> *Subject:* Re: [Talk-us] [Imports] Import WestCOG building footprints
>>>> in south-west Connecticut
>>>>
>>>>
>>>>
>>>> Hi Julien,
>>>>
>>>> Unfortunately, I have limited knowledge on the data quality as I wasn't
>>>> able to download it (the server returns error). I let the CT point of
>>>> contact (Scott) know about the problem -- he mentioned in our communication
>>>> that he forwarded the issue to the tech support team, but I haven't heard
>>>> from them since then and I'm still unable to download it.
>>>>
>>>>
>>>>
>>>> On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller <osm at lepiller.eu> wrote:
>>>>
>>>> So, it's been a week since that last message. Do you think we should
>>>> import addresses and buildings at the same time? Should we import the
>>>> buildings first and care about addresses later?
>>>>
>>>> Yury, what are your thoughts about the data source quality? Do you
>>>> think it's a good idea to import from WestCOG and maybe rely on CT data
>>>> for the rest of CT? I tried playing with the data and I didn't see any
>>>> difference between drawing the buildings from scratch and having to
>>>> simplify and correct CT's data.
>>>>
>>>> Thanks!
>>>>
>>>> Le Sat, 22 Aug 2020 19:36:23 -0400,
>>>> Martin Machyna <machyna at gmail.com> a écrit :
>>>>
>>>> > Thank Julien for pushing this forward!
>>>> >
>>>> > yeah, I tried to get addresses from here:
>>>> >
>>>> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
>>>> > and add them on top of the westCOG buildings.
>>>> >
>>>> > The data is a big mess because it's a join_table of like 30 different
>>>> > address databases. I lost a bit of motivation there, but I could have
>>>> > a look at it again.
>>>> >
>>>> > Martin
>>>> >
>>>> > On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller <osm at lepiller.eu>
>>>> > wrote:
>>>> >
>>>> > > Le Sat, 22 Aug 2020 13:30:02 -0400,
>>>> > > Yury Yatsynovich <yury.yatsynovich at gmail.com> a écrit :
>>>> > >
>>>> > > > Hi Julien,
>>>> > > > The following communication that I've had recently with a CT
>>>> > > > official might be of interest to you:
>>>> > > >
>>>> > > >
>>>> > >
>>>> > > Oh, great! I think we already saw this data (I tried to contact them
>>>> > > too, but never got a reply :/). From what we saw (I think it was in
>>>> > > February?) the footprints have simplification issues (see
>>>> > > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
>>>> > > instance) where they are too detailed, not square enough, etc. Some
>>>> > > buildings also have holes in them, when there's none in the imagery.
>>>> > >
>>>> > > So I think it's too bad to be used directly, without a lot of manual
>>>> > > effort to simplify, square and redraw the shapes. However, the
>>>> > > address data is very interesting, so maybe we could extract from
>>>> > > it? Or we could use a separate dataset if they have addresses
>>>> > > separately.
>>>> > >
>>>> > > _______________________________________________
>>>> > > Imports mailing list
>>>> > > Imports at openstreetmap.org
>>>> > > https://lists.openstreetmap.org/listinfo/imports
>>>> > >
>>>>
>>>>
>>>> _______________________________________________
>>>> Talk-us mailing list
>>>> Talk-us at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/talk-us
>>>>
>>>> _______________________________________________
>>>> Imports mailing list
>>>> Imports at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/imports
>>>>
>>> _______________________________________________
>>> Talk-us mailing list
>>> Talk-us at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk-us
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/imports/attachments/20200913/94a5d85a/attachment.htm>