[Talk-ca] Importing buildings in Canada
Nate Wessel
bike756 at gmail.com
Sun Jan 5 17:33:24 UTC 2020
John,
Those links were just to indicate what can be done with task setup. I
don't think we're close to setting up actual import tasks yet, though we
do seem to be on the road toward getting things sorted out :-)
Nate Wessel, PhD
Planner, Cartographer, Transport Nerd
NateWessel.com <https://www.natewessel.com>
On 2020-01-05 12:20 p.m., john whelan wrote:
> I'm getting lost as to who is organising what and who is taking
> responsibility for defining and setting up the tasks and where they
> are being set up.
>
> Are your two examples of what can be done? Or are they to be used by
> mappers to add buildings?
>
> My understanding was Daniel would identify those areas that looked the
> most interesting then these would be set up after the process to see
> if any local mappers were available and what their input would be.
>
> Thanks John
>
> On Sun, Jan 5, 2020, 11:41 AM Nate Wessel, <bike756 at gmail.com
> <mailto:bike756 at gmail.com>> wrote:
>
> The task size, and even shape is totally customizable. I've set up
> a couple that are entirely based on the density of the data:
> http://tasks.osmcanada.ca/project/168
> https://tasks.openstreetmap.us/project/107
>
> Best,
>
> Nate Wessel, PhD
> Planner, Cartographer, Transport Nerd
> NateWessel.com <https://www.natewessel.com>
>
> On 2020-01-04 12:40 p.m., Daniel @jfd553 wrote:
>>
>> Bonjour groupe
>>
>> Looks like we're going in the same direction so far :-)
>>
>> I agree with Nate regarding the implementation of the task
>> manager. In my experience, a size of a few blocks would be better
>> in urban areas, but boring in rural areas. Is it something that
>> can be adjusted?
>>
>> Daniel
>>
>> *From:*Nate Wessel [mailto:bike756 at gmail.com]
>> *Sent:* Saturday, January 04, 2020 10:09
>> *To:* talk-ca at openstreetmap.org <mailto:talk-ca at openstreetmap.org>
>> *Subject:* Re: [Talk-ca] Importing buildings in Canada
>>
>> Hi Daniel,
>>
>> Thank you for all the work you've put into this. I'd like to
>> offer a couple suggestions and/or clarifications for your
>> proposed import process, overview though it is.
>>
>> First, I think it is very important that a tasking manager is set
>> up on a city/by city basis only, and that only AFTER consensus is
>> achieved that the import should proceed in that area. I would
>> really like to avoid seeing the massive nationwide tasking that
>> was set up the first time around. We should be making it hard for
>> people to go rogue in regions where consensus for an import
>> doesn't (yet) exist.
>>
>> Related to this, though important enough to be a second point in
>> it's own right, the tasking squares need to be small enough that
>> a single user can manage them and inspect every single building
>> in a task. The first round of import used task squares that were
>> massive, and which couldn't be divided any further past a certain
>> point. Even in rural areas, it is likely inappropriate to import
>> areas larger than 1km^2. In central Toronto it would be (and was)
>> idiotic. An import that doesn't take local scale into account
>> shouldn't proceed. "Too big to load into JOSM" is about 100x too
>> big to import in my opinion and is not a good enough benchmark
>> for import batch sizing.
>>
>> That is, each import needs to be local, and not just in a
>> superficial sense.
>>
>> I'll also add that the issue of conflation doesn't seem to have
>> been worked out yet except to note that it is an issue. What will
>> we do with the millions of buildings which will substantially
>> overlap/duplicate existing buildings or imports? This needs to be
>> worked out in detail before anything starts up again.
>>
>> And what needs to be done about already existing low quality
>> imports? It's good to acknowledge their existence, but what will
>> be done about them? We've set up a task to clean up some of the
>> mess in Toronto ( http://tasks.osmcanada.ca/project/168 ) but
>> this is only the tip of the iceberg.
>>
>> Again, I thank everyone for their time and effort on this - we
>> can get this done if we go slow and do it right :-)
>>
>> Best,
>>
>> Nate Wessel, PhD
>> Planner, Cartographer, Transport Nerd
>> NateWessel.com <https://www.natewessel.com>
>>
>> On 2020-01-03 3:40 p.m., Daniel @jfd553 wrote:
>>
>> Bonjour groupe, mes excuses pour ce très long courriel !-)
>>
>> I have reviewed everything that has been written on the ODB
>> import (aka Canada Building Import) in Talk-ca and the wiki.
>> I proposed changes to some wiki pages (via talk tabs) to ease
>> the discussions about this import and the following. Now, in
>> order to restart the import, here are some thoughts and a
>> proposal on how to proceed to complete the task.
>>
>> *1. Issues with the ODB Data Import*
>>
>> Many concerns were raised about the import. One major concern
>> was to obtain local communities’ buy-in in the Canadian
>> context. Another concern was to improve the quality of the
>> data prior the import. The following paragraphs intend to
>> clear most of these concerns.
>>
>> *1.1. Which data import project?*
>>
>> According to the import guidelines (steps 3 & 4), a data
>> import explicitly refers to a single data source (ODB in our
>> case). Discussions about the availability and quality of
>> Microsoft or ESRI data, while interesting, are not relevant
>> as they should be dealt with as other import projects.
>>
>> *1.2. What has been imported so far?*
>>
>> According to what I found [1], the ODB import is completed
>> for 21 municipalities. These imports seem to have kept OSM
>> content’s history, at least for the samples checked, but many
>> problems were found. In some case, the imports brought
>> swimming pools in OSM because they were included in the
>> dataset (e.g. Moncton). In other cases, importing buildings
>> with accurate locations (XY) over content mapped from less
>> accurate imagery resulted in buildings that now overlap the
>> street network (e.g. Squamish). It means that all these 21
>> imports need to be carefully re-examined and corrected as
>> required.
>>
>> For 12 other municipalities, the import is partial, either
>> suspended as requested, or because previous imports had
>> already provided most of the buildings (often from the same
>> municipal provider). That said the import will definitely
>> improve OSM accuracy and completeness if done properly.
>>
>> *2. How should ODB Data be imported?*
>>
>> I will copy the following paragraphs in the “Canada Building
>> Import” wiki page [3] for a detailed discussion…
>>
>> Since the data (ODB, OSM and imagery) differ from one
>> municipality to another, there can be no imports at the
>> national or provincial level. We have to work on a municipal
>> basis and make sure to identify all the problems and the
>> corrective measures to apply when dealing with issues like
>> those I identified [1].
>>
>> *2.1 Importing Locally*
>>
>> According to the import guidelines (step 2), we must not
>> import the data without local buy-in. However, and contrarily
>> to some European country, there is usually no such thing as a
>> local OSM community in each municipality. However, we may
>> find a few local mappers from time to time. Working on a
>> municipal basis should allow identifying these local mappers
>> before doing the import. I often use this tool [2] to
>> identify and contact local mappers. Once identified, I
>> suggest that…
>>
>> - We contact them to explain our intents by referring to
>> appropriate wiki pages.
>>
>> - We wait a week or two to let them respond nothing, that
>> they have concerns, or wish to help.
>>
>> - Without negative answers we could proceed to the import.
>>
>> I first suggest that when a contributor wishes to import ODB
>> for a given municipality, he first identifies himself as
>> responsible for the import (we need to create an entry for
>> each municipality somewhere in the wiki). He can then contact
>> local mappers, as explain above, and go ahead with the import
>> once everything settled. For those who already made the
>> import, I suggest that they review their work since many
>> issues were detected with some of these imports.
>>
>> Since there are only a few local OSM communities in Canada,
>> and because Canada is large, I suggest not limiting the
>> import of a given municipality to the people of the concerned
>> province or region.
>>
>> *2.2 Pre-processing*
>>
>> Once local mappers have agreed, some pre-processing can be
>> done if required.
>>
>> A few months ago, I developed a tool that could be used to
>> process the data [4]. Concerns were raised because the
>> application was developed using proprietary software. So I
>> documented the whole process and algorithms in order to see
>> courageous coders converting it in open source software. In
>> the meantime, and as long as I have access to an FME licence,
>> I could process the data, when necessary, prior to make it
>> available through the task manager.
>>
>> Proposed pre-processing [4] includes:
>>
>> - Reading of original ODB data,
>>
>> - Removal of near collinear nodes (simplification),
>>
>> - Orthogonalization of buildings (for corners having near
>> right angles),
>>
>> - Tagging of building footprints,
>>
>> - Providing files in OSM format.
>>
>> /Proposed tagging:/ In addition to the tags produced by the
>> orthogonalization process [4] and the source tag (source
>> <https://wiki.openstreetmap.org/wiki/Key:source>=Statistics
>> Canada - Open Building Database), the name of the Census
>> Subdivision provided in ODB data [5] is used to add the
>> addr:city tag to each building.
>>
>> The pre-processing requires parameters that are specific to
>> the data to process. These parameters were estimated on a
>> municipal basis using actual ODB data. The processing time
>> increases exponentially according to the number of buildings
>> so, it may take a couple of days before the data is available
>> for a given municipality. Currently, the proposed
>> pre-processing does not convert terrace buildings into
>> individual houses nor it tags topological errors.
>>
>> *2.3. Import Process*
>>
>> After the local mappers, if any, agreed to the import, the
>> pre-processing completed when required, we can proceed to the
>> import.
>>
>> 1- Do not bulk import the data! Always use the task manager
>> (http://tasks.osmcanada.ca/). Select and open a task square
>> in JOSM. If it’s too big (e.g. too much work or request is
>> too big to load in JOSM), go back to the task manager and
>> split the task into smaller squares.
>>
>> 2- Load imagery layer (Bing or ESRI World Imagery) and align
>> the imagery with ODB data (i.e. create a new image offset) if
>> necessary because, unless proven otherwise, ODB should be
>> more accurate (XY) than most available images especially in
>> hilly areas.
>>
>> 3- Align the existing OSM content to the image (i.e. after
>> the new offset is applied) if required.
>>
>> 4- Currently step 2 and following as described in the wiki
>> [2]. I suggest merging the Conflation section [6] here and
>> reviewing everything to take into account the current proposal.
>>
>> *References*
>>
>> [1]
>> https://wiki.openstreetmap.org/wiki/The_Open_Database_of_Buildings
>>
>> [2]
>> https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Import_process
>>
>> [3] http://resultmaps.neis-one.org(“Overview of OpenStreetMap
>> Contributors aka who’s around me?”)
>>
>> [4] https://github.com/jfd553/OrthogonalizingBuildingFootprint
>>
>> [5]
>> https://www150.statcan.gc.ca/n1/pub/92-195-x/2011001/geo/csd-sdr/csd-sdr-eng.htm
>>
>> [6]
>> https://wiki.openstreetmap.org/wiki/Canada_Building_Import#Conflation
>>
>> Let’s move ahead!
>>
>> Daniel
>>
>>
>>
>> _______________________________________________
>>
>> Talk-ca mailing list
>>
>> Talk-ca at openstreetmap.org <mailto:Talk-ca at openstreetmap.org>
>>
>> https://lists.openstreetmap.org/listinfo/talk-ca
>>
> _______________________________________________
> Talk-ca mailing list
> Talk-ca at openstreetmap.org <mailto:Talk-ca at openstreetmap.org>
> https://lists.openstreetmap.org/listinfo/talk-ca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk-ca/attachments/20200105/0d00c63d/attachment-0001.htm>
More information about the Talk-ca
mailing list