[Osmf-talk] Mapping trees as buildings (was: Re: Alternative Strategic Plan)

Tue May 16 03:37:35 UTC 2023

Quick update here: Andy is right, this specific incident with the trees was
not part of any organized editing, the mapper was not a novice and likely
from the region. We discussed it a bit more on Telegram. Explicitly, the
quoted example is *not* part of any HOT or YouthMappers organized editing,
and this example in particular makes no sense talking about improving the
skill of human validators.

>From my initial comment, it was just an real world example of an individual
using RapID and AI generated suggestions (not an example of organized
editing, so by current practices, no need to review).

Looking at the photos, it's on the borderline of maybe be or not something,
but a human using other editors (without suggestions) very likely would
simply ignore. Also, the problem is not misaligned lines (like maybe we
would see on organized editing) but creating entire features that don't
exist which would be well aligned.

I don't know when RapiD was allowed to be used outside strictly trained
teams outside Facebook or even when the algorithms increased these kinds of
false positives which are harder to review, however the current situation
resembles a lot what happened in Egypt/India cited as reference on Wiki
page about the MapWithAI https://news.ycombinator.com/item?id=17856687 .
Some parts of the text of the link

> (...)
> Eventually we all gathered up and I just pushed the button to contribute
a random 1km / 1km square in Egypt (we'd already computed edits across the
country accounting for over 100% increase but things could always get
better). Then we waited. No one ever reached out so we started contributing
a little more at a steady rate to see if anyone else working in the country
noticed (including improving the original 1km / 1km box as the models
improved).
> (...)
> I ended up gathering a few too many people and myself along with the core
team was too hasty in communicating the quality bar for submission. We
planned to shadow edits as the week went on to make sure the new members
were up to speed but things unfolded much more quickly. Within a few hours
we accidentally submitted a few bad roads. A local mapper noticed as he'd
just driven the area on his motorcycle the day before. I immediately left a
meeting where we were negotiating buying more satellite imagery and jumped
on a bike back to the war room. No harm done, we had scripts ready to undo
edits.
> When I got into the war room things were more problematic. Quite a few of
the new folks had made similar mistakes so we paused everything. There was
now a small group of local mappers in an IRC channel worried about large
scale vandalism (though they quickly realized that wasn't the case). They
noticed the breadth of the edits and tracked down the accounts of most
everyone in the room. The map community in India is one of the better
communities but still a room of this many people making edits at a such a
scale was unlike anything you'd normally expect since the ML made editing
10-100x faster than hand tracing imagery.
> (...)

The rapid editor can be accessed on this link <https://rapideditor.org/>
(it is possible to preview suggestions without logging with OSM
credentials, so others here can check how bad it is in your region).
Despite past issues with MapWithAI being restricted to always require
validation, the current site has zero warnings for mappers (and we know
non-novices can upload the bad suggestions). The MapWithAI use license for
its users (link here https://mapwith.ai/doc/license/MapWithAILicense.pdf)
try to clearly assume no responsibility caused by its services (which would
be implied AI suggestions) in case user make any error, but the end data
would still be uploaded to OSM.

What to do?

Att.
Rocha

---------- Forwarded message ---------
De: john whelan <jwhelan0112 at gmail.com>
Date: seg, 15 de mai de 2023 19:25
Subject: Re: [Osmf-talk] Mapping trees as buildings (was: Re: Alternative
Strategic Plan)
To: Damilola Olufemi <olufemidamilola263 at gmail.com>
Cc: OSMF Talk <osmf-talk at openstreetmap.org>

Unfortunately that raises all sorts of questions.

First how do you know something has been validated.  Second what quality
standards do you suggest?  Lines up with Bing or other imagery?

Then you get to the quality of validation.  I've seen project managers in
HOT tag something as invalid when it wasn't.

HOT has done a lot of work on validation and even they struggle with the
subject.

I think the best approach is to say that the majority of mapping is
accurate and if anyone notices any problems they can correct it.

I think in general the first pass will probably be armchair mappers mapping
major highways, then HOT mapping settlements, followed locals taking
ownership and that's when you'll often see the quality improve.

There are bottom feeders such as myself that look for obvious errors, such
as a two km motorway connecting two small settlements, or a building mapped
three times and correct them but I don't mark anything in a positive way to
say I have formally validated it.

Cheerio John

On Mon, May 15, 2023, 17:45 Damilola Olufemi <olufemidamilola263 at gmail.com>
wrote:

> Hello,
>
> I had a similar experience with a Remote Sensing organization here in
> Nigeria. I was overwhelmed with questions that I was not prepared enough to
> give answers to about data quality on OpenStreetMap. My suggestion would be
> that imageries on OpenStreetMap get reviewed more often and be sure that
> the imageries are up to date with a good resolution. I once mentioned that
> data quality on OpenStreetMap is easily questionable, considering the fact
> that it is being contributed by volunteers that have other activities to
> focus on irrespective of the amount of training received. Only a few will
> be keen on delivering the best. Perhaps volunteers could map while trained
> staffs validate. My opinion.
>
> Regards
> Damilola
>
> On Mon, May 15, 2023 at 12:30 PM Andy Townsend <ajt1047 at gmail.com> wrote:
>
>> On 15/05/2023 05:47, Emerson Rocha via osmf-talk wrote:
>> > Let me give a real world example. Weeks ago on unofficial
>> > OpenStreetMap Telegram channel  (message
>> > https://t.me/OpenStreetMapOrg/101372 ) a mapper from Colombia
>> > complained that the buildings added in an area (near volcano Nevado
>> > del Ruiz) was visited by local civil defense. The building=yes (added
>> > by a human mapper, but geometry suggested
>> > Microsoft/buildingFootprints) actually was a... tree. Guayacanes and
>> > Yarumos to be more exact. He was pissed off on the chat, saying it
>> > already complained in the past.
>>
>>
>> For the avoidance of doubt this was a mistake by a human mapper
>> apparently from the same country (and with a bit of experience - 1000
>> edits or so) using "RapiD 1.1.9".  It wasn't a new contributor to OSM
>> attending a mapathon organised by an NGO abroad.  There appear to be no
>> changeset comments on their changes suggesting that people have noticed
>> problems with "trees as buildings" or similar.  If a complaint was made,
>> it doesn't appear to have been made to this mapper.  Put bluntly,
>> they're not going to know how problematic tools like "RapiD" can be
>> unless someone tells them.  That's not to say that RapiD aren't really
>> useful at finding missing buildings; just that they will also find some
>> false positives too (a quick scan locally (UK) finds around 20 missing
>> sheds and small agricultural buildings that no-one has bothered to add -
>> and one that appears to be a bit of scrap metal, that isn't.
>>
>> I'd suggest (and will suggest on Telegram) that a changeset comment
>> helping them understand how to use tools such as this would be the best
>> initial approach.
>>
>> Best Regards,
>>
>> Andy
>>
>> (as usual writing here in a personal capacity, and apologies from
>> dragging this thread even further from "strategic" things)
>>
>>
>>
>> _______________________________________________
>> osmf-talk mailing list
>> osmf-talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osmf-talk
>>
> _______________________________________________
> osmf-talk mailing list
> osmf-talk at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmf-talk
>
_______________________________________________
osmf-talk mailing list
osmf-talk at openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmf-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/osmf-talk/attachments/20230516/6dd7a0d2/attachment.htm>