[Imports] [Talk-us] Import WestCOG building footprints in south-west Connecticut

Martin Machyna machyna at gmail.com
Sun Sep 13 02:51:20 UTC 2020


Thanks for catching the 'null' values. In geopandas data.frame each
row/object must have assigned values for all fields. I was just assuming
null values don't get exported.
I guess this can be fixed with something like: sed 's/, "[^"]*": null//g'

How did you keep track of your import progress? Did you use taskscheduler?
We are looking for something that, but without public access.

On Fri, Sep 11, 2020 at 8:04 AM TC Haddad <tchaddad at gmail.com> wrote:

>
> Hi Martin, I have limited experience with imports, but I helped in
> Portland, OR. I looked at your Westport file Two things I would pass on:
>
> - if the field is null, I don't think the tag should be in the import.
> Null fields make the files unnecessarily large and provide no value
> - we removed the buildings that conflicted with existing buildings in
> advance
> - we had a separate process to handle buildings with multiple addresses
> - we tiled the geography into squares of no more than 500 buildings each.
> This makes it possible for an import volunteer to have a workflow where
> they import a square and then review every building for the conflicts you
> mentioned (roads crossing buildings, etc). More buildings imported at one
> time makes the review very difficult, especially if there are lots of such
> corrections
>
> Our Github was here in case there is anything else of value to you
> https://github.com/pdxosgeo/pdxbldgimport, and our wiki page was here:
> https://wiki.openstreetmap.org/wiki/Portland,_OR_Bldg_import
>
> On Thu, Sep 10, 2020 at 6:31 PM Martin Machyna <machyna at gmail.com> wrote:
>
>> So to update this thread, I have integrated addresses from CT Open Data
>> dataset and also updated the wiki page (
>> https://wiki.openstreetmap.org/wiki/Connecticut/Western_COG_Building_Import
>> )
>>
>> The whole dataset can be looked at:
>> https://drive.google.com/file/d/10hUl09WSmK-I8h1hkMlELIGwDYsEziSe/view?usp=sharing
>> For quick loading I also made a subset for one town (Westport):
>> https://drive.google.com/file/d/1oMSbHXpPY5eLSHhGlikGy2uudyq3VyoV/view?usp=sharing
>>
>> If there is any issue, please let us know.
>>
>> On the side note of the CT Open Data buildings suitability for import I
>> found a way how to simplify all buildings in an automated way in python.
>> Here is a quick comparison of before/after
>> https://files.slack.com/files-pri/T029HV94T-F01AD0FNFD3/simplified.png
>> (you need to have Slack account)
>> It doesn't look so bad and we could consider it for a next round of
>> import.
>>
>> Just for a future reference in case someone would need to do the same,
>> the python code is:
>>
>> import geopandas as gpd
>> import pandas as pd
>> from shapely import speedups
>> speedups.enable()
>>
>> address =
>> gpd.read_file("Connecticut_Buildings_with_Addresses_experimental.shp")
>> simple = address.simplify(0.000005, preserve_topology=True)
>> simple.to_file('Buildings-simplified.geojson', driver='GeoJSON')
>>
>>
>>
>> On Sat, Aug 29, 2020 at 6:50 PM <joe.sapletal at charter.net> wrote:
>>
>>> I was going to look at the buildings too.  I’ve used a tool in ArcGIS to
>>> correct some pretty awful buildings, but I couldn’t download them either.
>>>   If there is no hurry, I’d check in again with the contact on Monday.  It
>>> would be nice to have the buildings with addresses on them.
>>>
>>>
>>>
>>> Joe
>>>
>>>
>>>
>>> *From:* Yury Yatsynovich <yury.yatsynovich at gmail.com>
>>> *Sent:* Saturday, August 29, 2020 5:05 PM
>>> *To:* Julien Lepiller <osm at lepiller.eu>
>>> *Cc:* imports at openstreetmap.org; talk-us at openstreetmap.org
>>> *Subject:* Re: [Talk-us] [Imports] Import WestCOG building footprints
>>> in south-west Connecticut
>>>
>>>
>>>
>>> Hi Julien,
>>>
>>> Unfortunately, I have limited knowledge on the data quality as I wasn't
>>> able to download it (the server returns error). I let the CT point of
>>> contact (Scott) know about the problem -- he mentioned in our communication
>>> that he forwarded the issue to the tech support team, but I haven't heard
>>> from them since then and I'm still unable to download it.
>>>
>>>
>>>
>>> On Sat, Aug 29, 2020, 4:57 PM Julien Lepiller <osm at lepiller.eu> wrote:
>>>
>>> So, it's been a week since that last message. Do you think we should
>>> import addresses and buildings at the same time? Should we import the
>>> buildings first and care about addresses later?
>>>
>>> Yury, what are your thoughts about the data source quality? Do you
>>> think it's a good idea to import from WestCOG and maybe rely on CT data
>>> for the rest of CT? I tried playing with the data and I didn't see any
>>> difference between drawing the buildings from scratch and having to
>>> simplify and correct CT's data.
>>>
>>> Thanks!
>>>
>>> Le Sat, 22 Aug 2020 19:36:23 -0400,
>>> Martin Machyna <machyna at gmail.com> a écrit :
>>>
>>> > Thank Julien for pushing this forward!
>>> >
>>> > yeah, I tried to get addresses from here:
>>> >
>>> http://geodata-ctmaps.opendata.arcgis.com/datasets/bfa7da83da384c2aa809882179369dc4_0/features/305004
>>> > and add them on top of the westCOG buildings.
>>> >
>>> > The data is a big mess because it's a join_table of like 30 different
>>> > address databases. I lost a bit of motivation there, but I could have
>>> > a look at it again.
>>> >
>>> > Martin
>>> >
>>> > On Sat, Aug 22, 2020 at 2:19 PM Julien Lepiller <osm at lepiller.eu>
>>> > wrote:
>>> >
>>> > > Le Sat, 22 Aug 2020 13:30:02 -0400,
>>> > > Yury Yatsynovich <yury.yatsynovich at gmail.com> a écrit :
>>> > >
>>> > > > Hi Julien,
>>> > > > The following communication that I've had recently with a CT
>>> > > > official might be of interest to you:
>>> > > >
>>> > > >
>>> > >
>>> > > Oh, great! I think we already saw this data (I tried to contact them
>>> > > too, but never got a reply :/). From what we saw (I think it was in
>>> > > February?) the footprints have simplification issues (see
>>> > > https://files.slack.com/files-pri/T029HV94T-FTDGDHXTM/image.png for
>>> > > instance) where they are too detailed, not square enough, etc. Some
>>> > > buildings also have holes in them, when there's none in the imagery.
>>> > >
>>> > > So I think it's too bad to be used directly, without a lot of manual
>>> > > effort to simplify, square and redraw the shapes. However, the
>>> > > address data is very interesting, so maybe we could extract from
>>> > > it? Or we could use a separate dataset if they have addresses
>>> > > separately.
>>> > >
>>> > > _______________________________________________
>>> > > Imports mailing list
>>> > > Imports at openstreetmap.org
>>> > > https://lists.openstreetmap.org/listinfo/imports
>>> > >
>>>
>>>
>>> _______________________________________________
>>> Talk-us mailing list
>>> Talk-us at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk-us
>>>
>>> _______________________________________________
>>> Imports mailing list
>>> Imports at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/imports
>>>
>> _______________________________________________
>> Talk-us mailing list
>> Talk-us at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk-us
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/imports/attachments/20200912/ac9a838c/attachment.htm>


More information about the Imports mailing list