[OSM-talk] Announcing Daylight Map Distribution

[Michal Migurski]

Hi Joseph, thanks for your questions!

We are unsure yet about the public release schedule we want to commit to, because a lot of it depends on community feedback which we’re getting now. I hope we will do this again!

We do correct OSM upstream for the errors that we find. When our human review process catches a map error, we do two things:

1) Hold it back from release to our display maps 
2) Fix the error upstream in OSM.org

Ideally, the fix subsequently passes our process on the next round.

-mike.

> On Mar 9, 2020, at 5:47 PM, Joseph Eisenberg <joseph.eisenberg at gmail.com> wrote:
> 
> Interesting! That sounds like a huge amount of work, if you are
> validating every changeset in the globe, though so far there is no
> schedule to release this frequently.
> 
> Thank you for following the Open Database license by releasing this to
> the public.
> 
> Is facebook planning to create new planet files like this on a
> frequent basis, for internal use? If so, those should be released to
> the public on a regular schedule, if I understand the ODbL correctly.
> 
> Can I confirm that facebook is editing Openstreetmap data to fix the
> errors which are found? Or is this only happening in some cases?
> 
> The diary post gave links that explain how this was made (both from
> September 2019):
> - https://engineering.fb.com/ml-applications/mars/ "MaRS: How
> Facebook keeps maps current and accurate"
> 
> "We calculate the absolute diff between the two OSM versions (from
> option 1 above).
> We then split this diff into a set of LoChas that can be individually
> applied (from option 2)."
> ...
> "At a high level, a map change is vetted by a reviewer,"
> 
> - https://2019.stateofthemap.us/program/sun/keepin-it-fresh-and-good-continuous-ingestion-of-osm-data-at-facebook.html
> "“Keepin’ it fresh (and good)!” - Continuous Ingestion of OSM Data at
> Facebook":
> 
> This is a video presentation, so I haven't seen it all, but the
> abstract says: "we have created an automated ingestion and integrity
> framework for OSM data that allows us to selectively update parts of
> the map instead of doing a full snapshot change all at once.
> 
> "Decomposing a large set of changes in this way gives us the
> flexibility to rapidly ingest our own additions to the map, focus on
> geographical areas of importance to downstream products, and allows us
> to quickly apply hotfixes whenever egregious problems do arise.
> 
> "With millions of tiny changes happening every week, we have created a
> system that is built on per-feature approval and preprocessing, that
> allows us to ingest changes at scale, while creating rules to
> automatically process logical changesets and enforce integrity
> constraints (e.g. anti-vandalism, anti-profanity etc.).
> 
> "Due to the contextual nature of some of the changes in OpenStreetMap,
> the system combines Human Approval, necessary for highly visible
> features such as names of large administrative areas, with Automated
> AI/ML-based approval: for example, using computer vision techniques to
> reconcile newly created features against satellite imagery ground
> truth, or applying NLP techniques to determine whether other
> user-visible string changes are sensible and valid. These components
> are combined to create a continuous ingest-validate-deploy cycle for
> OSM map data."
> 
> Lot's of buzz words there! But it sounds like it is a combination of
> computer algorithms and human checking for vandalism and errors.
> 
> - Joseph Eisenberg
> 
> On 3/10/20, Michal Migurski <mike at teczno.com> wrote:
>> Hi everyone,
>> 
>> I’m writing to let you know about a new OpenStreetMap project Facebook just
>> released. It’s called Daylight Map Distribution. Daylight is a complete,
>> downloadable preview of OpenStreetMap data we plan to start using in a
>> number of our public maps:
>> 
>> 	https://www.openstreetmap.org/user/migurski/diary/392416
>> 
>> Facebook uses maps to let our users find friends, businesses, groups and
>> more. OpenStreetMap (OSM) has a substantial global footprint of map data
>> built and maintained by a dedicated community of global mappers and it’s a
>> natural choice for us. Every day, OSM receives millions of contributions
>> from the community. Some of these contributions may have intentional and
>> unintentional edits that are incompatible with our needs. Our mapping teams
>> work to scrub these contributions for consistency and quality.
>> 
>> What’s Included in the Daylight Map Distribution:
>> 
>> 	• A PBF planet file composed of 100% OSM data, released under the terms of
>> the Open Database License.
>> 	• Only those edits which have been validated to contain no malicious
>> vandalism or unintentional errors so we can show them in our display maps
>> 
>> This is just an initial first release, and we’re looking for feedback from
>> the community to decide what would be useful to release in the future and
>> how frequently. I’d be interested to hear any response you might have about
>> it!
>> 
>> -mike.
>> 
>> ----------------------------------------------------------------
>> michal migurski- contact info and pgp key:
>> sf/ca            http://mike.teczno.com/contact.html
>> 
>> 
>> _______________________________________________
>> talk mailing list
>> talk at openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>> 
> 

----------------------------------------------------------------
michal migurski- contact info and pgp key:
sf/ca            http://mike.teczno.com/contact.html