[ML4OSM] Measure machine learning assisted mapping

[Tech 4.Good]

Hi Felix and Team,

Very interesting study your team is about to conduct.

It will indeed be very interesting to see the evaluation of its direct capabilities to identify buildings and roads, and the comparison between the speed/efforts involved in the different scenarios that are outlined.

Below, I have tried to list some of the ideas that came up while going through your post.


Evaluation of support for specific activities + adding confidence scores to support evaluation
If possible, it will also be very interesting to see an evaluation of how the AI-assisted scenarios, both stand-alone and human-assisted, are able to:

  *   Identify populated areas and estimate population size/density within a given area, also evaluated in relation to effort/speed
  *   Identify transport routes between populated areas, requiring that roads are correctly identified as connected, that also in relation to effort/speed

100% accuracy in neither AI-assisted nor human-only mapping can be expected, but it would be very interesting to see if its accuracy can be evaluated as “good enough” or “not good enough” for activities such as the above. The terms ”good enough/not good enough” are of course subjective definitions, but it would be interesting to understand whether the models are close or far away from being valid for activities such as the above. True/false accuracy may give some indications, but if possible, it would also be interesting to see confidence scores of the predictions, as this can also help to understand if some issues could be overcome by further training of the models.


Enabler/prerequisite for further AI-assisted developments
Also, it would be interesting to understand the aspect of whether the AI-assisted mapping in its current state, can be an enabler or potentially is a prerequisite for future AI-assisted developments. Hence if possible, it would be interesting to understand if the given functionality potentially is a prerequisite for being able to:

  *   Develop recognition of new features: Evaluate if there is a potential in reusing/retraining existing models/to map new features, for example waterways, bridges, etc.
  *   Identify land use/land cover: Evaluate potential in reusing/retraining existing models or create new models to automate analysis of land use/land cover
  *   Support time series analysis to understand changes over time: Evaluate potential to use the AI-assisted mapping to analyze changes over time, for example to compare current mappings to maps of the same areas before or after certain events or time periods. Examples of purpose could be to compare areas before and after the event of natural disasters, changes due to climate change, analysis of urban development or migration over time, etc. Potentially, it could be evaluated if the existing AI-assisted mapping could improve the process for automating these types of time series analyses?

Evaluate support for “new mappers” vs “advanced mappers/validators”
Additionally, to further understand the impact of accuracy and speed, it could be helpful to understand this both from the perspective of “new mappers” and “advanced mappers/validators”. The AI-assisted mapping may provide benefits and challenges in different ways, for instance be helpful to new mappers in certain situations, for example indicating what to look for, while new mappers may also need additional support to understand how to correct invalid automated mapping. Likewise, for the advanced mappers/validators, different pros/cons could be foreseen. It could be helpful to have a list with different hypotheses to be confirmed/rejected in connection with the evaluation.


Potential need for more validators/advanced mappers
One hypothesis to add, could be that given that the AI-assisted mapping provides a potential to map more projects/areas, how would this influence the need and availability for more validators/advanced mappers, people who have reached a high level of experience through long-term practices within mapping. Can it be evaluated how more people can be mobilized to help validate, and if AI-assisted mapping can help people reach the ability to validate faster or attract more people to pursue validation?


It has been really great to see the results already achieved by your team, it certainly will make a great impact, and already provides great inspiration to how ML and image recognition can be applied for the cause. I hope the above can help spark some ideas and will be happy to support.


Best regards, Anders



---------- Forwarded message ---------
From: Felix Delattre via machine-learning <machine-learning at openstreetmap.org<mailto:machine-learning at openstreetmap.org>>
Date: Tue, 2 Jun 2020 at 09:25
Subject: [ML4OSM] Measure machine learning assisted mapping
To: <machine-learning at openstreetmap.org<mailto:machine-learning at openstreetmap.org>>
Cc: Felix Delattre <felix-lists at delattre.de<mailto:felix-lists at delattre.de>>


Dear all,

As you probably know, the conversation about machine learning techniques
and their use for OpenStreetMap has been very emotional in our
community. Opinions range from the potential negative impacts this could
have, to the hope that it would significantly improve the quality and
also the speed of OSM mapping, because it allows people to focus on what
they do best.

We want to take an evidence-based look at the effects of machine
learning mapping on OpenStreetMap. To do this we are working together
with several organizations (German Geoscience Research Center,
University of Heidelberg and the OpenStreetMap humanitarian team) to
conduct research that will quantify the measurable impact of the
currently proposed mapping workflow. We believe that a reproducible and
transparent study will give us a clue.

We are planning to do an experiment comparing four different datasets
from the same area:

* Reference data: Well mapped OpenStreetMap (over a longer period of time)
* Conventional remote mapping data from OpenStreetMap (single mapping event)
* Machine learning assisted remote mapping data with RapID (single
mapping event)
* Data created by the latest generation of AI models (without any editing)

And we want to look at the indicators around it:

* Quality: descriptive analysis of objects
* The speed of mapping

We want all the data and workflows we produce to be as reusable as
possible. For this reason, all data from this experiment will be open
and transparent.

Please contact us if you are interested in any further analysis, we are
happy to hear your suggestions before we start, so that we can ensure
that all the raw data are as useful and correct as possible.

In any case, we will keep you informed here.

Best wishes,
Felix

PD: There is a related blog post:
https://www.hotosm.org/updates/how-we-measure-the-effects-of-ai-assisted-mapping/

--
machine-learning mailing list
machine-learning at openstreetmap.org<mailto:machine-learning at openstreetmap.org>
https://lists.openstreetmap.org/listinfo/machine-learning
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/machine-learning/attachments/20200608/b3c1be66/attachment.htm>