[Talk-ca] 2020 building import wiki comment by Nate Wessel

Fri Jan 18 21:37:03 UTC 2019

I'm on board with removing redundant nodes, as long as it doesn't affect
actual geometries much.
Also there are quite a few duplicated nodes in buildings. They can be
removed in JOSM in one click before upload but better to do it at the
source.

On Fri, Jan 18, 2019 at 4:25 PM James <james2432 at gmail.com> wrote:

> I can run all the shapefiles through qgis simplify tool if this resolves
> the issue...
>
> On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel <bike756 at gmail.com wrote:
>
>> With default settings in JOSM, sure. In the import I was working on, we
>> used a Douglas-Peucker algorithm with a 20cm threshold (before the import
>> started) and it worked beautifully. We had many points that seemed to have
>> been introduced in the shapefiles as some kind of data artifact - they
>> didn't add any detail to the shape at all. This procedure removed almost
>> all of them with no discernible reduction in quality.
>> Nate Wessel
>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>> NateWessel.com <http://natewessel.com>
>>
>> On 1/18/19 4:03 PM, James wrote:
>>
>> dare you to run simplify tool on anything remotely round, it will make it
>> look like garbage
>>
>> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan <jwhelan0112 at gmail.com
>> wrote:
>>
>>> The import mailing list was pointed to the correct page of the wiki.
>>> The initial post was to say this is what we were thinking of and there was
>>> a comment saying we needed to change the comment line.
>>>
>>> >There is no mention of this proposed import on the import catalogue
>>>
>>>
>>> The import process was reviewed by the person who set up the Ottawa
>>> import did we miss that step on the Ottawa import as well?  Neither was it
>>> raised as a concern on the import mailing list. I think this is very minor
>>> and can be corrected.
>>>
>>> We learnt a fair bit on the Ottawa import and my expectation is since we
>>> are using experienced mappers to do the import conflation would be either
>>> handled by them or the building not imported. We aren't using new mappers
>>> in a mapathon here and with experienced mappers then I think you have to
>>> trust them.  The world isn't perfect. Think in terms of service level.
>>>
>>> >There are 2X more nodes than needed to represent the building
>>> accurately.
>>>
>>> The problem with correcting this is you are introducing approximations.
>>> This will vary according to the source and this can be simplified or
>>> corrected once its in OSM. I think this is a different issue of a
>>> mechanical edit that needs to be considered separately.
>>>
>>> If we are concerned with database size then I suggest we change the
>>> instructions to say put the source comment on the change set rather than on
>>> the building outline.
>>>
>>> Cheerio John
>>>
>>>
>>> Nate Wessel wrote on 2019-01-18 3:06 PM:
>>>
>>> John,
>>>
>>> You seem to be playing the long game with this data - it sounds like
>>> you've been working with this a lot longer than I have, and you've put in
>>> the time and effort to help make this actually-quite-incredible dataset
>>> available to us. I don't want to stop the import from happening - quite the
>>> opposite. I just want to make sure that the time is taken to do this right.
>>> OSM deserves that. Your (our) long awaited victory will be the sweeter for
>>> our patience now.
>>>
>>> There are several specific issues I see where the I's are not crossed,
>>> nor the t's dotted. I've mentioned several already, so I'll try to be brief
>>> (I really need to get back to working on my dissertation).
>>>
>>> 1) There was extremely limited discussion on the imports mailing list.
>>> The initial email did not make clear the scope of the project. I read the
>>> email and did not think twice at it, thinking it was entirely about Ottawa.
>>> The link in that email was actually to the Ottawa import, and not this one,
>>> which seems to have been only in draft at the time.
>>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
>>> As such, this project has NOT been reviewed by the imports list, which
>>> is a requirement for proceeding with the import.
>>>
>>> 2) There is no mention of this proposed import on the import catalogue (
>>> https://wiki.openstreetmap.org/wiki/Import/Catalogue)
>>> which is required in the imports guidelines. I suspect many other
>>> guidelines have not been followed.
>>>
>>> 3) The wiki page describing the import is not adequate to assess the
>>> quality of the data or of the proposed import. See for example:
>>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
>>> The import guidelines call for a description of how conflation will be
>>> handled. The fact that two of the major importers seem to have a
>>> substantial disagreement about how to handle existing data indicates this
>>> was not well discussed and I can see that it isn't well documented.
>>>
>>> 4) The buildings need to be simplified, quite a bit actually. Most
>>> buildings have multiple nodes representing straight lines. This bloats the
>>> database and makes things harder to edit by hand later. There are probably
>>> 2x more nodes than are needed to represent the data accurately, making it
>>> harder for editors and data consumers to work with down the road.This is a
>>> simple fix that will save countless hours later.
>>>
>>> ... I could go on, but I think this is plenty sufficient to justify
>>> pressing pause on all this.
>>>
>>> Again, I don't in any way want to disrespect the work that has gone into
>>> this effort already. We're all volunteers here and I know how much time
>>> this all takes. However. importing all/most of the buildings in Canada is a
>>> monstrously large task, which will have to dance around a lot of people's
>>> toes. We should expect this to take a really damn long time if we're going
>>> to do it right. We need to have the patience to learn from experience, from
>>> critique, and from the wisdom of the people who've learned from flawed
>>> imports in the past and have devised guidelines and processes so that we
>>> can have better experiences with this in the future.
>>> Nate Wessel
>>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>>> NateWessel.com <http://natewessel.com>
>>>
>>> On 1/18/19 2:24 PM, john whelan wrote:
>>>
>>> My background is I'm a retired civil servant who has written and
>>> overseen procurement documents and fairly large procurements.  Dotting the
>>> is and crossing the Ts are my speciality.
>>>
>>> There are two parts to an import.  The first part is the part played by
>>> the import mailing group.  They confine themselves to is the license
>>> correct and do you have a reasonable plan.  In this case the license is one
>>> of the few that has been confirmed by the Legal Working Group of
>>> OpenStreetMap and as such no questions were raised about it on the import
>>> mailing list.  We have methodology that has been used before successfully
>>> with the Ottawa building outline import. There were major discussions both
>>> on talk-ca and the import mailing group before that import took place and
>>> we took note of the issues raised and addressed them.  The licensing issue
>>> goes back about eight years to when I was talking to Federal Government
>>> Treasury Board and explaining their Open Data license did not align with
>>> OSM.  That is why their license is now known as 2.0.
>>>
>>> The second part is the local group makes the decision to import they are
>>> the authority no one else.
>>>
>>> Apparently you were not part of the talk-ca when the discussions took
>>> place which would have been the time and place to raise concerns.
>>>
>>> When the Ottawa import was done there were one or two places where the
>>> existing buildings and the import overlapped.  In the instructions on the
>>> import there are instructions to cover this. Specifically there is a
>>> validation step.  I seem to recall the error rate was of the order of 1%
>>> and I expect this latest batch to be roughly the same.
>>>
>>> If you can identify which municipalities data is of poor quality then
>>> I'm sure we can remove these.  For the most part these are from the
>>> foundation plans recorded by the municipality using professional surveying
>>> techniques.
>>>
>>> Would you like to clarify exactly where I failed to dot the Is and cross
>>> the Ts please.
>>>
>>> Many Thanks
>>>
>>> John
>>>
>>>
>>>
>>> On Fri, 18 Jan 2019 at 13:37, Nate Wessel <bike756 at gmail.com> wrote:
>>>
>>>> Hi John,
>>>>
>>>> As Steve has said, you seem to be the only one suggesting that
>>>> thousands of import committees might need to be formed. Certainly I'm not
>>>> suggesting that.
>>>>
>>>> My understanding of OSM import procedure (and wiki-style projects more
>>>> generally) is that imports should operate in an essentially consensual way
>>>> where possible. The goal is to build consent and bring people on board with
>>>> a project or a change by addressing their concerns in a meaningful and
>>>> respectful way.
>>>>
>>>> I think that I have made some substantive and troubling claims about
>>>> the quality of the data being imported. I've pointed out that this project
>>>> has not followed the import procedures that were produced by a community of
>>>> mappers larger than just those in Canada.
>>>>
>>>> So to respond to your implication, I am in some sense the one reviewing
>>>> the project, just as I would welcome you to find ways that my own
>>>> contributions could be better. If you want my credentials for reviewing
>>>> your work, here they are:
>>>>
>>>> 1) I am an active contributor to OSM in Toronto, where I live (and
>>>> elsewhere)
>>>>
>>>> 2) I am currently helping to lead a building import in Hamilton County
>>>> Ohio that has better addressed some of the issues I see this import
>>>> struggling with. I can help you do the same.
>>>>
>>>> 3) I've been doing research in GIS for a long time now, though I don't
>>>> need that to tell you that the issues I've described are hardly
>>>> insurmountable technically or even all that difficult to fix. It would take
>>>> maybe one day's hard work to get the technical side of this right.
>>>>
>>>> I think Canadian OSMers will agree that we can take a pause to get
>>>> things right on such a massive import. If they don't - if I'm shouted down
>>>> or better, if my critiques are adequately addressed, then I will leave you
>>>> to finish the project in peace. I might even lend a hand if all goes well,
>>>> as I sincerely hope it does :-)
>>>>
>>>> Best,
>>>> Nate Wessel
>>>> Jack of all trades, Master of Geography, PhD candidate in Urban Planning
>>>> NateWessel.com <http://natewessel.com>
>>>>
>>>> On 1/18/19 1:11 PM, john whelan wrote:
>>>>
>>>> I know of no other way to contact him but he made an interesting
>>>> comment that the project is on hold in the wiki pending review.
>>>>
>>>> Would he care to comment on who is supposed to be reviewing the project?
>>>>
>>>> My understanding is that the import was raised in talk-ca before it
>>>> commenced for comment and these were generally favourable.  I took that as
>>>> the local mappers to Canada had been consulted and they are the "local
>>>> mappers" authority in this case.
>>>>
>>>> I understand he has concerns about local mappers making decisions but
>>>> in Canada we have been importing similar data through CANVEC for some
>>>> time.  CANVEC data comes from a number of sources including municipal data.
>>>>
>>>> Is he suggesting that each of the 3,700 municipalities in Canada should
>>>> form a group of local mappers who can make individual decisions on whether
>>>> their municipal data should be imported and we should end up with 3,700
>>>> import plans?
>>>>
>>>> Thanks John
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Talk-ca mailing listTalk-ca at openstreetmap.orghttps://lists.openstreetmap.org/listinfo/talk-ca
>>>>
>>>> _______________________________________________
>>>> Talk-ca mailing list
>>>> Talk-ca at openstreetmap.org
>>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>>
>>>
>>> --
>>> Sent from Postbox
>>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>
>>> _______________________________________________
>>> Talk-ca mailing list
>>> Talk-ca at openstreetmap.org
>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>
>> _______________________________________________
> Talk-ca mailing list
> Talk-ca at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk-ca
>

-- 
Best Regards,
          Yaro Shkvorets
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20190118/bcf945d5/attachment-0001.html>