[OSM-talk] Announcing Daylight Map Distribution

[Warin]

Fixing stuff in OSM purely from imagery may not be good.

A local mapper who sees something may add it before any satellite imagery has it.

If you then 'fix' this back to the satellite imagery you will have committed an error,
and that error may dissuade our most important resource from ever making any further changes- the local mapper.

Be very careful!


On 10/3/20 1:11 pm, Michal Migurski wrote:
> Hi Joseph, thanks for your questions!
>
> We are unsure yet about the public release schedule we want to commit to, because a lot of it depends on community feedback which we’re getting now. I hope we will do this again!
>
> We do correct OSM upstream for the errors that we find. When our human review process catches a map error, we do two things:
>
> 1) Hold it back from release to our display maps
> 2) Fix the error upstream in OSM.org
>
> Ideally, the fix subsequently passes our process on the next round.
>
> -mike.
>
>> On Mar 9, 2020, at 5:47 PM, Joseph Eisenberg <joseph.eisenberg at gmail.com> wrote:
>>
>> Interesting! That sounds like a huge amount of work, if you are
>> validating every changeset in the globe, though so far there is no
>> schedule to release this frequently.
>>
>> Thank you for following the Open Database license by releasing this to
>> the public.
>>
>> Is facebook planning to create new planet files like this on a
>> frequent basis, for internal use? If so, those should be released to
>> the public on a regular schedule, if I understand the ODbL correctly.
>>
>> Can I confirm that facebook is editing Openstreetmap data to fix the
>> errors which are found? Or is this only happening in some cases?
>>
>> The diary post gave links that explain how this was made (both from
>> September 2019):
>> - https://engineering.fb.com/ml-applications/mars/ "MaRS: How
>> Facebook keeps maps current and accurate"
>>
>> "We calculate the absolute diff between the two OSM versions (from
>> option 1 above).
>> We then split this diff into a set of LoChas that can be individually
>> applied (from option 2)."
>> ...
>> "At a high level, a map change is vetted by a reviewer,"
>>
>> - https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html
>> "“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at
>> Facebook":
>>
>> This is a video presentation, so I haven't seen it all, but the
>> abstract says: "we have created an automated ingestion and integrity
>> framework for OSM data that allows us to selectively update parts of
>> the map instead of doing a full snapshot change all at once.
>>
>> "Decomposing a large set of changes in this way gives us the
>> flexibility to rapidly ingest our own additions to the map, focus on
>> geographical areas of importance to downstream products, and allows us
>> to quickly apply hotfixes whenever egregious problems do arise.
>>
>> "With millions of tiny changes happening every week, we have created a
>> system that is built on per-feature approval and preprocessing, that
>> allows us to ingest changes at scale, while creating rules to
>> automatically process logical changesets and enforce integrity
>> constraints (e.g. anti-vandalism, anti-profanity etc.).
>>
>> "Due to the contextual nature of some of the changes in OpenStreetMap,
>> the system combines Human Approval, necessary for highly visible
>> features such as names of large administrative areas, with Automated
>> AI/ML-based approval: for example, using computer vision techniques to
>> reconcile newly created features against satellite imagery ground
>> truth, or applying NLP techniques to determine whether other
>> user-visible string changes are sensible and valid. These components
>> are combined to create a continuous ingest-validate-deploy cycle for
>> OSM map data."
>>
>> Lot's of buzz words there! But it sounds like it is a combination of
>> computer algorithms and human checking for vandalism and errors.
>>
>> - Joseph Eisenberg
>>
>> On 3/10/20, Michal Migurski <mike at teczno.com> wrote:
>>> Hi everyone,
>>>
>>> I’m writing to let you know about a new OpenStreetMap project Facebook just
>>> released. It’s called Daylight Map Distribution. Daylight is a complete,
>>> downloadable preview of OpenStreetMap data we plan to start using in a
>>> number of our public maps:
>>>
>>> 	https://www.openstreetmap.org/user/migurski/diary/392416
>>>
>>> Facebook uses maps to let our users find friends, businesses, groups and
>>> more. OpenStreetMap (OSM) has a substantial global footprint of map data
>>> built and maintained by a dedicated community of global mappers and it’s a
>>> natural choice for us. Every day, OSM receives millions of contributions
>>> from the community. Some of these contributions may have intentional and
>>> unintentional edits that are incompatible with our needs. Our mapping teams
>>> work to scrub these contributions for consistency and quality.
>>>
>>> What’s Included in the Daylight Map Distribution:
>>>
>>> 	• A PBF planet file composed of 100% OSM data, released under the terms of
>>> the Open Database License.
>>> 	• Only those edits which have been validated to contain no malicious
>>> vandalism or unintentional errors so we can show them in our display maps
>>>
>>> This is just an initial first release, and we’re looking for feedback from
>>> the community to decide what would be useful to release in the future and
>>> how frequently. I’d be interested to hear any response you might have about
>>> it!
>>>
>>> -mike.
>>>
>>> ----------------------------------------------------------------
>>> michal migurski- contact info and pgp key:
>>> sf/ca            http://mike.teczno.com/contact.html
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20200310/474d8051/attachment-0001.htm>