[Talk-ca] 2020 building import wiki comment by Nate Wessel
Nate Wessel
bike756 at gmail.com
Fri Jan 18 21:30:15 UTC 2019
I'm not familiar with the tool, but that is essentially what I'm asking
for - nothing all that complicated. We would need to make sure we're
not losing any valuable detail though, and ensure that topology is
preserved where buildings share nodes.
Nate Wessel
Jack of all trades, Master of Geography, PhD candidate in Urban Planning
NateWessel.com <http://natewessel.com>
On 1/18/19 4:24 PM, James wrote:
> I can run all the shapefiles through qgis simplify tool if this
> resolves the issue...
>
> On Fri., Jan. 18, 2019, 4:08 p.m. Nate Wessel <bike756 at gmail.com
> <mailto:bike756 at gmail.com> wrote:
>
> With default settings in JOSM, sure. In the import I was working
> on, we used a Douglas-Peucker algorithm with a 20cm threshold
> (before the import started) and it worked beautifully. We had many
> points that seemed to have been introduced in the shapefiles as
> some kind of data artifact - they didn't add any detail to the
> shape at all. This procedure removed almost all of them with no
> discernible reduction in quality.
>
> Nate Wessel
> Jack of all trades, Master of Geography, PhD candidate in Urban
> Planning
> NateWessel.com <http://natewessel.com>
>
> On 1/18/19 4:03 PM, James wrote:
>> dare you to run simplify tool on anything remotely round, it will
>> make it look like garbage
>>
>> On Fri., Jan. 18, 2019, 3:49 p.m. John Whelan
>> <jwhelan0112 at gmail.com <mailto:jwhelan0112 at gmail.com> wrote:
>>
>> The import mailing list was pointed to the correct page of
>> the wiki. The initial post was to say this is what we were
>> thinking of and there was a comment saying we needed to
>> change the comment line.
>>
>> >There is no mention of this proposed import on the import
>> catalogue
>>
>>
>> The import process was reviewed by the person who set up the
>> Ottawa import did we miss that step on the Ottawa import as
>> well? Neither was it raised as a concern on the import
>> mailing list. I think this is very minor and can be corrected.
>>
>> We learnt a fair bit on the Ottawa import and my expectation
>> is since we are using experienced mappers to do the import
>> conflation would be either handled by them or the building
>> not imported. We aren't using new mappers in a mapathon here
>> and with experienced mappers then I think you have to trust
>> them. The world isn't perfect. Think in terms of service level.
>>
>> >There are 2X more nodes than needed to represent the
>> building accurately.
>>
>> The problem with correcting this is you are introducing
>> approximations. This will vary according to the source and
>> this can be simplified or corrected once its in OSM. I think
>> this is a different issue of a mechanical edit that needs to
>> be considered separately.
>>
>> If we are concerned with database size then I suggest we
>> change the instructions to say put the source comment on the
>> change set rather than on the building outline.
>>
>> Cheerio John
>>
>>
>> Nate Wessel wrote on 2019-01-18 3:06 PM:
>>>
>>> John,
>>>
>>> You seem to be playing the long game with this data - it
>>> sounds like you've been working with this a lot longer than
>>> I have, and you've put in the time and effort to help make
>>> this actually-quite-incredible dataset available to us. I
>>> don't want to stop the import from happening - quite the
>>> opposite. I just want to make sure that the time is taken to
>>> do this right. OSM deserves that. Your (our) long awaited
>>> victory will be the sweeter for our patience now.
>>>
>>> There are several specific issues I see where the I's are
>>> not crossed, nor the t's dotted. I've mentioned several
>>> already, so I'll try to be brief (I really need to get back
>>> to working on my dissertation).
>>>
>>> 1) There was extremely limited discussion on the imports
>>> mailing list. The initial email did not make clear the scope
>>> of the project. I read the email and did not think twice at
>>> it, thinking it was entirely about Ottawa. The link in that
>>> email was actually to the Ottawa import, and not this one,
>>> which seems to have been only in draft at the time.
>>> https://lists.openstreetmap.org/pipermail/imports/2018-November/005812.html
>>> As such, this project has NOT been reviewed by the imports
>>> list, which is a requirement for proceeding with the import.
>>>
>>> 2) There is no mention of this proposed import on the import
>>> catalogue (https://wiki.openstreetmap.org/wiki/Import/Catalogue)
>>> which is required in the imports guidelines. I suspect many
>>> other guidelines have not been followed.
>>>
>>> 3) The wiki page describing the import is not adequate to
>>> assess the quality of the data or of the proposed import.
>>> See for example:
>>> https://wiki.openstreetmap.org/wiki/WikiProject_Canada/Canada_Stats_Canada_Building_Outlines_Import/Plan#Risks
>>> The import guidelines call for a description of how
>>> conflation will be handled. The fact that two of the major
>>> importers seem to have a substantial disagreement about how
>>> to handle existing data indicates this was not well
>>> discussed and I can see that it isn't well documented.
>>>
>>> 4) The buildings need to be simplified, quite a bit
>>> actually. Most buildings have multiple nodes representing
>>> straight lines. This bloats the database and makes things
>>> harder to edit by hand later. There are probably 2x more
>>> nodes than are needed to represent the data accurately,
>>> making it harder for editors and data consumers to work with
>>> down the road.This is a simple fix that will save countless
>>> hours later.
>>>
>>> ... I could go on, but I think this is plenty sufficient to
>>> justify pressing pause on all this.
>>>
>>> Again, I don't in any way want to disrespect the work that
>>> has gone into this effort already. We're all volunteers here
>>> and I know how much time this all takes. However. importing
>>> all/most of the buildings in Canada is a monstrously large
>>> task, which will have to dance around a lot of people's
>>> toes. We should expect this to take a really damn long time
>>> if we're going to do it right. We need to have the patience
>>> to learn from experience, from critique, and from the wisdom
>>> of the people who've learned from flawed imports in the past
>>> and have devised guidelines and processes so that we can
>>> have better experiences with this in the future.
>>>
>>> Nate Wessel
>>> Jack of all trades, Master of Geography, PhD candidate in
>>> Urban Planning
>>> NateWessel.com <http://natewessel.com>
>>>
>>> On 1/18/19 2:24 PM, john whelan wrote:
>>>> My background is I'm a retired civil servant who has
>>>> written and overseen procurement documents and fairly large
>>>> procurements. Dotting the is and crossing the Ts are my
>>>> speciality.
>>>>
>>>> There are two parts to an import. The first part is the
>>>> part played by the import mailing group. They confine
>>>> themselves to is the license correct and do you have a
>>>> reasonable plan. In this case the license is one of the
>>>> few that has been confirmed by the Legal Working Group of
>>>> OpenStreetMap and as such no questions were raised about it
>>>> on the import mailing list. We have methodology that has
>>>> been used before successfully with the Ottawa building
>>>> outline import. There were major discussions both on
>>>> talk-ca and the import mailing group before that import
>>>> took place and we took note of the issues raised and
>>>> addressed them. The licensing issue goes back about eight
>>>> years to when I was talking to Federal Government Treasury
>>>> Board and explaining their Open Data license did not align
>>>> with OSM. That is why their license is now known as 2.0.
>>>>
>>>> The second part is the local group makes the decision to
>>>> import they are the authority no one else.
>>>>
>>>> Apparently you were not part of the talk-ca when the
>>>> discussions took place which would have been the time and
>>>> place to raise concerns.
>>>>
>>>> When the Ottawa import was done there were one or two
>>>> places where the existing buildings and the import
>>>> overlapped. In the instructions on the import there are
>>>> instructions to cover this. Specifically there is a
>>>> validation step. I seem to recall the error rate was of
>>>> the order of 1% and I expect this latest batch to be
>>>> roughly the same.
>>>>
>>>> If you can identify which municipalities data is of poor
>>>> quality then I'm sure we can remove these. For the most
>>>> part these are from the foundation plans recorded by the
>>>> municipality using professional surveying techniques.
>>>>
>>>> Would you like to clarify exactly where I failed to dot the
>>>> Is and cross the Ts please.
>>>>
>>>> Many Thanks
>>>>
>>>> John
>>>>
>>>>
>>>>
>>>> On Fri, 18 Jan 2019 at 13:37, Nate Wessel
>>>> <bike756 at gmail.com <mailto:bike756 at gmail.com>> wrote:
>>>>
>>>> Hi John,
>>>>
>>>> As Steve has said, you seem to be the only one
>>>> suggesting that thousands of import committees might
>>>> need to be formed. Certainly I'm not suggesting that.
>>>>
>>>> My understanding of OSM import procedure (and
>>>> wiki-style projects more generally) is that imports
>>>> should operate in an essentially consensual way where
>>>> possible. The goal is to build consent and bring people
>>>> on board with a project or a change by addressing their
>>>> concerns in a meaningful and respectful way.
>>>>
>>>> I think that I have made some substantive and troubling
>>>> claims about the quality of the data being imported.
>>>> I've pointed out that this project has not followed the
>>>> import procedures that were produced by a community of
>>>> mappers larger than just those in Canada.
>>>>
>>>> So to respond to your implication, I am in some sense
>>>> the one reviewing the project, just as I would welcome
>>>> you to find ways that my own contributions could be
>>>> better. If you want my credentials for reviewing your
>>>> work, here they are:
>>>>
>>>> 1) I am an active contributor to OSM in Toronto, where
>>>> I live (and elsewhere)
>>>>
>>>> 2) I am currently helping to lead a building import in
>>>> Hamilton County Ohio that has better addressed some of
>>>> the issues I see this import struggling with. I can
>>>> help you do the same.
>>>>
>>>> 3) I've been doing research in GIS for a long time now,
>>>> though I don't need that to tell you that the issues
>>>> I've described are hardly insurmountable technically or
>>>> even all that difficult to fix. It would take maybe one
>>>> day's hard work to get the technical side of this right.
>>>>
>>>> I think Canadian OSMers will agree that we can take a
>>>> pause to get things right on such a massive import. If
>>>> they don't - if I'm shouted down or better, if my
>>>> critiques are adequately addressed, then I will leave
>>>> you to finish the project in peace. I might even lend a
>>>> hand if all goes well, as I sincerely hope it does :-)
>>>>
>>>> Best,
>>>>
>>>> Nate Wessel
>>>> Jack of all trades, Master of Geography, PhD candidate
>>>> in Urban Planning
>>>> NateWessel.com <http://natewessel.com>
>>>>
>>>> On 1/18/19 1:11 PM, john whelan wrote:
>>>>> I know of no other way to contact him but he made an
>>>>> interesting comment that the project is on hold in the
>>>>> wiki pending review.
>>>>>
>>>>> Would he care to comment on who is supposed to be
>>>>> reviewing the project?
>>>>>
>>>>> My understanding is that the import was raised in
>>>>> talk-ca before it commenced for comment and these were
>>>>> generally favourable. I took that as the local
>>>>> mappers to Canada had been consulted and they are the
>>>>> "local mappers" authority in this case.
>>>>>
>>>>> I understand he has concerns about local mappers
>>>>> making decisions but in Canada we have been importing
>>>>> similar data through CANVEC for some time. CANVEC
>>>>> data comes from a number of sources including
>>>>> municipal data.
>>>>>
>>>>> Is he suggesting that each of the 3,700 municipalities
>>>>> in Canada should form a group of local mappers who can
>>>>> make individual decisions on whether their municipal
>>>>> data should be imported and we should end up with
>>>>> 3,700 import plans?
>>>>>
>>>>> Thanks John
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Talk-ca mailing list
>>>>> Talk-ca at openstreetmap.org <mailto:Talk-ca at openstreetmap.org>
>>>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>> _______________________________________________
>>>> Talk-ca mailing list
>>>> Talk-ca at openstreetmap.org
>>>> <mailto:Talk-ca at openstreetmap.org>
>>>> https://lists.openstreetmap.org/listinfo/talk-ca
>>>>
>>
>> --
>> Sent from Postbox
>> <https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>
>> _______________________________________________
>> Talk-ca mailing list
>> Talk-ca at openstreetmap.org <mailto:Talk-ca at openstreetmap.org>
>> https://lists.openstreetmap.org/listinfo/talk-ca
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20190118/8f8f8634/attachment-0001.html>
More information about the Talk-ca
mailing list