[Talk-ca] 2020 building import wiki comment by Nate Wessel

Fri Jan 18 21:24:10 UTC 2019

I can run all the shapefiles through qgis simplify tool if this resolves
the issue...

On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel <bike756 at gmail.com wrote:

> With default settings in JOSM, sure. In the import I was working on, we
> used a Douglas-Peucker algorithm with a 20cm threshold (before the import
> started) and it worked beautifully. We had many points that seemed to have
> been introduced in the shapefiles as some kind of data artifact - they
> didn't add any detail to the shape at all. This procedure removed almost
> all of them with no discernible reduction in quality.
> Nate Wessel
> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
> NateWessel.com <http://natewessel.com>
>
> On 1/18/19 4:03 PM, James wrote:
>
> dare you to run simplify tool on anything remotely round, it will make it
> look like garbage
>
> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan <jwhelan0112 at gmail.com
> wrote:
>
>> The import mailing list was pointed to the correct page of the wiki.  The
>> initial post was to say this is what we were thinking of and there was a
>> comment saying we needed to change the comment line.
>>
>> >There is no mention of this proposed import on the import catalogue
>>
>>
>> The import process was reviewed by the person who set up the Ottawa
>> import did we miss that step on the Ottawa import as well?  Neither was it
>> raised as a concern on the import mailing list. I think this is very minor
>> and can be corrected.
>>
>> We learnt a fair bit on the Ottawa import and my expectation is since we
>> are using experienced mappers to do the import conflation would be either
>> handled by them or the building not imported. We aren't using new mappers
>> in a mapathon here and with experienced mappers then I think you have to
>> trust them.  The world isn't perfect. Think in terms of service level.
>>
>> >There are 2X more nodes than needed to represent the building accurately.
>>
>> The problem with correcting this is you are introducing approximations.
>> This will vary according to the source and this can be simplified or
>> corrected once its in OSM. I think this is a different issue of a
>> mechanical edit that needs to be considered separately.
>>
>> If we are concerned with database size then I suggest we change the
>> instructions to say put the source comment on the change set rather than on
>> the building outline.
>>
>> Cheerio John
>>
>>
>> Nate Wessel wrote on 2019-01-18 3:06 PM:
>>
>> John,
>>
>> You seem to be playing the long game with this data - it sounds like
>> you've been working with this a lot longer than I have, and you've put in
>> the time and effort to help make this actually-quite-incredible dataset
>> available to us. I don't want to stop the import from happening - quite the
>> opposite. I just want to make sure that the time is taken to do this right.
>> OSM deserves that. Your (our) long awaited victory will be the sweeter for
>> our patience now.
>>
>> There are several specific issues I see where the I's are not crossed,
>> nor the t's dotted. I've mentioned several already, so I'll try to be brief
>> (I really need to get back to working on my dissertation).
>>
>> 1) There was extremely limited discussion on the imports mailing list.
>> The initial email did not make clear the scope of the project. I read the
>> email and did not think twice at it, thinking it was entirely about Ottawa.
>> The link in that email was actually to the Ottawa import, and not this one,
>> which seems to have been only in draft at the time.
>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
>> As such, this project has NOT been reviewed by the imports list, which is
>> a requirement for proceeding with the import.
>>
>> 2) There is no mention of this proposed import on the import catalogue (
>> https://wiki.openstreetmap.org/wiki/Import/Catalogue)
>> which is required in the imports guidelines. I suspect many other
>> guidelines have not been followed.
>>
>> 3) The wiki page describing the import is not adequate to assess the
>> quality of the data or of the proposed import. See for example:
>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
>> The import guidelines call for a description of how conflation will be
>> handled. The fact that two of the major importers seem to have a
>> substantial disagreement about how to handle existing data indicates this
>> was not well discussed and I can see that it isn't well documented.
>>
>> 4) The buildings need to be simplified, quite a bit actually. Most
>> buildings have multiple nodes representing straight lines. This bloats the
>> database and makes things harder to edit by hand later. There are probably
>> 2x more nodes than are needed to represent the data accurately, making it
>> harder for editors and data consumers to work with down the road.This is a
>> simple fix that will save countless hours later.
>>
>> ... I could go on, but I think this is plenty sufficient to justify
>> pressing pause on all this.
>>
>> Again, I don't in any way want to disrespect the work that has gone into
>> this effort already. We're all volunteers here and I know how much time
>> this all takes. However. importing all/most of the buildings in Canada is a
>> monstrously large task, which will have to dance around a lot of people's
>> toes. We should expect this to take a really damn long time if we're going
>> to do it right. We need to have the patience to learn from experience, from
>> critique, and from the wisdom of the people who've learned from flawed
>> imports in the past and have devised guidelines and processes so that we
>> can have better experiences with this in the future.
>> Nate Wessel
>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>> NateWessel.com <http://natewessel.com>
>>
>> On 1/18/19 2:24 PM, john whelan wrote:
>>
>> My background is I'm a retired civil servant who has written and overseen
>> procurement documents and fairly large procurements.  Dotting the is and
>> crossing the Ts are my speciality.
>>
>> There are two parts to an import.  The first part is the part played by
>> the import mailing group.  They confine themselves to is the license
>> correct and do you have a reasonable plan.  In this case the license is one
>> of the few that has been confirmed by the Legal Working Group of
>> OpenStreetMap and as such no questions were raised about it on the import
>> mailing list.  We have methodology that has been used before successfully
>> with the Ottawa building outline import. There were major discussions both
>> on talk-ca and the import mailing group before that import took place and
>> we took note of the issues raised and addressed them.  The licensing issue
>> goes back about eight years to when I was talking to Federal Government
>> Treasury Board and explaining their Open Data license did not align with
>> OSM.  That is why their license is now known as 2.0.
>>
>> The second part is the local group makes the decision to import they are
>> the authority no one else.
>>
>> Apparently you were not part of the talk-ca when the discussions took
>> place which would have been the time and place to raise concerns.
>>
>> When the Ottawa import was done there were one or two places where the
>> existing buildings and the import overlapped.  In the instructions on the
>> import there are instructions to cover this. Specifically there is a
>> validation step.  I seem to recall the error rate was of the order of 1%
>> and I expect this latest batch to be roughly the same.
>>
>> If you can identify which municipalities data is of poor quality then I'm
>> sure we can remove these.  For the most part these are from the foundation
>> plans recorded by the municipality using professional surveying techniques.
>>
>> Would you like to clarify exactly where I failed to dot the Is and cross
>> the Ts please.
>>
>> Many Thanks
>>
>> John
>>
>>
>>
>> On Fri, 18 Jan 2019 at 13:37, Nate Wessel <bike756 at gmail.com> wrote:
>>
>>> Hi John,
>>>
>>> As Steve has said, you seem to be the only one suggesting that thousands
>>> of import committees might need to be formed. Certainly I'm not suggesting
>>> that.
>>>
>>> My understanding of OSM import procedure (and wiki-style projects more
>>> generally) is that imports should operate in an essentially consensual way
>>> where possible. The goal is to build consent and bring people on board with
>>> a project or a change by addressing their concerns in a meaningful and
>>> respectful way.
>>>
>>> I think that I have made some substantive and troubling claims about the
>>> quality of the data being imported. I've pointed out that this project has
>>> not followed the import procedures that were produced by a community of
>>> mappers larger than just those in Canada.
>>>
>>> So to respond to your implication, I am in some sense the one reviewing
>>> the project, just as I would welcome you to find ways that my own
>>> contributions could be better. If you want my credentials for reviewing
>>> your work, here they are:
>>>
>>> 1) I am an active contributor to OSM in Toronto, where I live (and
>>> elsewhere)
>>>
>>> 2) I am currently helping to lead a building import in Hamilton County
>>> Ohio that has better addressed some of the issues I see this import
>>> struggling with. I can help you do the same.
>>>
>>> 3) I've been doing research in GIS for a long time now, though I don't
>>> need that to tell you that the issues I've described are hardly
>>> insurmountable technically or even all that difficult to fix. It would take
>>> maybe one day's hard work to get the technical side of this right.
>>>
>>> I think Canadian OSMers will agree that we can take a pause to get
>>> things right on such a massive import. If they don't - if I'm shouted down
>>> or better, if my critiques are adequately addressed, then I will leave you
>>> to finish the project in peace. I might even lend a hand if all goes well,
>>> as I sincerely hope it does :-)
>>>
>>> Best,
>>> Nate Wessel
>>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>>> NateWessel.com <http://natewessel.com>
>>>
>>> On 1/18/19 1:11 PM, john whelan wrote:
>>>
>>> I know of no other way to contact him but he made an interesting comment
>>> that the project is on hold in the wiki pending review.
>>>
>>> Would he care to comment on who is supposed to be reviewing the project?
>>>
>>> My understanding is that the import was raised in talk-ca before it
>>> commenced for comment and these were generally favourable.  I took that as
>>> the local mappers to Canada had been consulted and they are the "local
>>> mappers" authority in this case.
>>>
>>> I understand he has concerns about local mappers making decisions but in
>>> Canada we have been importing similar data through CANVEC for some time.
>>> CANVEC data comes from a number of sources including municipal data.
>>>
>>> Is he suggesting that each of the 3,700 municipalities in Canada should
>>> form a group of local mappers who can make individual decisions on whether
>>> their municipal data should be imported and we should end up with 3,700
>>> import plans?
>>>
>>> Thanks John
>>>
>>>
>>>
>>> _______________________________________________
>>> Talk-ca mailing listTalk-ca at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/talk-ca
>>>
>>> _______________________________________________
>>> Talk-ca mailing list
>>> Talk-ca at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>
>>
>> --
>> Sent from Postbox
>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>
>> _______________________________________________
>> Talk-ca mailing list
>> Talk-ca at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk-ca
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20190118/af1ed246/attachment-0001.html>