<div dir="ltr"><div>Hi Drishtie,</div><div><br></div><div>thanks for your extensive reply. In general it looks good to me. One point however. Earlier in this thread I mentioned: ' I would be more interested to know whether community buy-in is okay. I haven't seen a response from the Thai community on this posting. I think they should be responding in this discussion before any import.'</div><div><br></div><div>Is the local Thai community involved? (still miss them in this thread)</div><div><br></div><div>Kind regards, Johan</div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-03-21 22:02 GMT+01:00 osm <span dir="ltr"><<a href="mailto:osm@fb.com" target="_blank">osm@fb.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" vlink="#954F72" link="#0563C1" bgcolor="white">
<div class="m_7937096899910185766WordSection1">
<p class="m_7937096899910185766MsoPlainText">Hello OSM Enthusiasts,<br>
<br>
Firstly, thank you for all the feedback. We really appreciate it. Sorry for the delay in responding. I have spent the last few days gathering all the questions and have come up with a list of answers that I hope brings some clarity. I tried to send this e-mail
yesterday but had too many example pics so it didn’t go through. Please click on the “Example” links to see pictures we shared to help clarify some of the question around process.
<span style="font-family:Wingdings">J</span><br>
<br>
<b><span style="color:#0070c0">Can you share DG imagery?</span></b><br>
Our license does not currently permit this. However, to reiterate what Kevin from Digital Globe said, earlier on this thread, this will not be an issue in a few weeks when all of OSM will get an imagery refresh.
<br>
<br>
<b><span style="color:#0070c0">Facebook License with Digital Globe</span></b><br>
This license is necessary to comply with our obligations to Digital Globe, and we think the benefits of supplying this data outweigh the licensing restraints. The data will be under ODBL once it has been contributed to OSM in compliance with the terms.<br>
<br>
There is no contradiction between the OSM terms and the Facebook license with Digital Globe. Our license terms do not apply to data once contributed to OSM and termination of the agreement with Digital Globe would not impact the data that has already been contributed
to OSM. <br>
<br>
<b><span style="color:#0070c0">Is the Imagery free of Cloud Cover?</span></b><br>
The intended output of +Vivid is cloud free, and in most cases, DG meets this. There are some highly cloudy parts of the world where it’s unavoidable. On average though, this product maintains less than 1% cloud cover.
<br>
<br>
<b><span style="color:#0070c0">Who are your editors?</span></b><br>
Our team consists of 3 engineers and 14 editors, and usernames can be found on the Import wiki. All editors have also updated their profile to say they map for Facebook. You can see their names on their profile. All our mappers are qualified for editing with
backgrounds in GIS, geography and are extremely technically savvy. We have also spent a considerable amount of time over the last 5 months training specifically for OSM.
<br>
<br>
The mappers are not bound by time limits and metrics as the focus is primarily on quality. We do however track time taken to edit, density of tile and who mapped what tile to use this information to plan for future mapping.
<br>
<br>
Facebook does not control these accounts and all users manage their own profiles and passwords. Once we start mapping we also expect each mapper to respond to questions from the community. All feedback is discussed as a group to ensure we are all aware of the
feedback and can learn as team. No other accounts other than the ones listed here and the wiki have been used to make edits for Facebook.
<br>
<br>
Our team includes:<br>
<a href="https://www.openstreetmap.org/user/VLD001" target="_blank">Jaclyn</a> - VLD001 <br>
<a href="https://www.openstreetmap.org/user/VLD002" target="_blank">Annie</a> - VLD002<br>
<a href="https://www.openstreetmap.org/user/VLD003" target="_blank">Alexandra</a> - VLD003<br>
<a href="https://www.openstreetmap.org/user/VLD004" target="_blank">Sheffield</a> - VLD004 <br>
<a href="https://www.openstreetmap.org/user/VLD005" target="_blank">Mei</a> - VLD005<br>
<a href="https://www.openstreetmap.org/user/VLD006" target="_blank">Joseph</a> - VLD006<br>
<a href="https://www.openstreetmap.org/user/VLD007" target="_blank">Kurt</a> - VLD007<br>
<a href="https://www.openstreetmap.org/user/VLD008" target="_blank">Pablo Presti-Mohr</a> VLD008<br>
<a href="https://www.openstreetmap.org/user/RVR001" target="_blank">Goerge</a> - RVR001<br>
<a href="https://www.openstreetmap.org/user/RVR002" target="_blank">Jeff</a> - RVR002<br>
<a href="https://www.openstreetmap.org/user/RVR003" target="_blank">David</a> - RVR003<br>
<a href="https://www.openstreetmap.org/user/RVR004" target="_blank">Yunzhi</a> - RVR004<br>
<a href="https://www.openstreetmap.org/user/RVR005" target="_blank">Adrian</a> - RVR005<br>
<a href="https://www.openstreetmap.org/user/RVR006" target="_blank">Stefani</a> - RVR006<br>
<a href="https://www.openstreetmap.org/user/RVR007" target="_blank">Mohamed</a> - RVR007<br>
<br>
<b><span style="color:#0070c0">Who is behind <a href="mailto:osm@fb.com" target="_blank"><span style="color:#0070c0">osm@fb.com</span></a>?</span></b><br>
There are 4 people behind this email. This helps us collaborate as a team and is the most efficient way for everyone to be on the same page. More people also ensures that no emails are missed. I (Drishtie Patel) as the Program Manager primarily watch this email
and respond to queries. The other 3 people are our Engineers namely, Ming Gao and Saikat Basu and the Product Manager for Maps at Facebook, Sadi Khan.
<br>
<br>
<b><span style="color:#0070c0">Can the Chageset be more specific?</span></b><br>
Yes we can absolutely change that to import=facebook-ai-____. We don't specifically have versions but can include the area or task number for example. Open to suggestions.<br>
<br>
As for adding more details to our edits: there were no changes to current OSM data in this sample and we plan to add notes when we upload as would normally happen when using the Tasking Manager and iD. The sample shared has not been uploaded yet
<span style="font-family:Wingdings">J</span><br>
<br>
<b><span style="color:#0070c0">Can you Publish Source Code for ML</span></b><br>
We cannot share the source code for Machine Learning at this time, but we can and will share our internal tools.<br>
<br>
<b><span style="color:#0070c0">Can you Publish Source Code for internal tools?</span></b><br>
Yes. We are currently using a version of <a href="https://github.com/openstreetmap/iD" target="_blank">
iD Editor</a> and the <a href="https://github.com/hotosm/osm-tasking-manager2" target="_blank">HOT Tasking Manager</a>. We are absolutely going to share our tooling. We are currently working on creating a Github where we will post soon.<br>
<br>
<b><span style="color:#0070c0">Can Facebook Share other data?</span></b><br>
This is clear violation of privacy so we cannot share this kind of information. We are exploring ways to crowdsource road names for OSM and will share that as we have a more concrete plan.
<br>
<br>
<b><span style="color:#0070c0">Are you going to Map other countries?</span></b><br>
Yes. We are moving slowly to focus efforts on one country at a time to make sure our process is accepted by the OSM community out of respect for the process.
<br>
<br>
<b><span style="color:#0070c0">Yes we made a mistake</span></b><br>
We've learned from a number of missteps in our initial efforts in Egypt and Thailand last year. We have since refined our processes, hired a team of well experienced mappers and are working to follow all the processes and requests being asked of us.
<br>
<br>
<b><span style="color:#0070c0">What is the Overall Process?</span></b><br>
<b>PHASE 1 - Generating Road Masks </b><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Training data is created by editors. <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16781235_1170760256386350_6655657355411193856_n.jpg?oh=2a84c16359c99bec01f9d1d23b9241ee&oe=595CF8FC" target="_blank">
Example</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Training data used by engineers in ML to produce road masks.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17365429_1592481874110291_3999181234856001536_n.png?oh=df901589c651edc1883bf0472eb1d4ca&oe=59670662" target="_blank">
Example</a> <u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Road masks are processed to remove low confidence predictions and add connections between short breaks.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"><b> </b><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"><b>PHASE 2 - Creating Road Vectors (.osm files)</b><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Road masks are then processed using an algorithm from a black and white image to a vector format specifically a .osm file.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16781103_1267857896629755_8804482991712632832_n.png?oh=cb25f91d0e6159b08f6ec342353ae77a&oe=593F3547" target="_blank">
Example</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Our .osm file is then merged with current OSM changesets. During this process the following things take place<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>All current OSM data for an area is merged, keeping full history.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>No changes are made to the current OSM data.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>Duplicated roads are deleted from the ML generated data. Current OSM data is always taken over facebook generated data.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>New ML identified roads are connected to current OSM roads.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>We then have a locally stored file containing both our generated roads and the current OSM data set for a specific area.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"><br>
<b>PHASE 3 - Human Validation</b><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>We use the Tasking manager to divide up tasks and create a flow of editing and validation for the Facebook mapping team.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17365380_1448559118508805_536801470968758272_n.jpg?oh=3cb38653fa004f2a20eeaabc2cfa583c&oe=5971703D" target="_blank">
Example</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>The Mapper picks a task and loads the locally stored .osm file generated from the post processing steps outlined above.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17433416_1595081350521384_2351112230801506304_n.jpg?oh=304c8670744ee2baf27c78043cc6943a&oe=595ECAF0" target="_blank">
Example</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>The task is opened up in our enhanced version of iD. (This includes both the roads generated by us and the current OSM data).
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16686335_1118257054950682_8271679018604953600_n.jpg?oh=b033e8c22c7b7248a1def85d1733fadc&oe=596EA8D8" target="_blank">
Example</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>4.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Our generated roads are highlighted in a different color so editors can inspect it for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fix these issues manually one
by one, using Digital Globe satellite imagery background.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>5.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>We do not delete current OSM data, but make typical OSM editing changes where necessary to ensure high quality of the data (more precisely follow imagery, consistent tagging, etc).<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>6.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Our modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint <a href="https://github.com/osmlab/osmlint/" target="_blank">https://github.com/osmlab/<wbr>osmlint/</a>. This allows us to check for quality and conflicts with current OSM data. Some examples of what we check
for include:<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>isolated roads that are not connected to rest of the map.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16781243_1457031487701473_198753992489041920_n.jpg?oh=a3413fae5079f7ce38519d9977b28a9b&oe=5971B86C" target="_blank">
Example-Before</a>. <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17433411_1244723028968011_867537480591605760_n.jpg?oh=f5a7cbeebec559fc6c80bfbfeef2d1b6&oe=59255F3A" target="_blank">
Example-After</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>roads crossing waterways.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>roads with repeating node IDs.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:1.0in">
<u></u><span style="font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u>roads with possible connections after a short extension.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17365393_1561733437200741_5520973609088057344_n.jpg?oh=486151bca24ba30880738c15a1770f87&oe=596A8307" target="_blank">
Example-Before</a>. <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17351734_1279516578799625_2079794241860534272_n.jpg?oh=5eb3c2118ae35ab40ddae75f225c7ad8&oe=595C2079" target="_blank">
Example-After</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>7.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>In case of conflicts between our newly created roads and other OSM editor's mapping, our mapper will almost always choose “keep their edits”. If we think our newly added roads are better aligned with the latest version of DG satellite
imagery, we'll contact the other mapper offline to reach resolution. If we end up changing other mapper's edits, we will leave detailed notes to explain why.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>8.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Until all errors are fixed mappers will not be able to save. Here is<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16685255_1430090083699471_1532498821462884352_n.png?oh=8438b1e92957b809f59b874507521bbc&oe=5973316B" target="_blank"> an example</a>
of a highlighted issue prompting the mapper to fix it.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>9.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>After fixing all the issues detected, our mapper clicks the “Save Local” button in the iD tool to save their edited roads locally for validation.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>10.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>A second person, the validator then goes into the same task to verify that the data looks correct, makes necessary changes as needed and clicks “Submit” to finally upload the tile to OSM.<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"><br>
<b><span style="color:#0070c0">Can you share more data samples?</span></b><br>
We have attached a few samples that we are sharing based on the 3 Phases explained above. We do these in small batches and are not sitting on a bunch of data for the whole country but are happy to share what we have created over the last couple of days.
<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"> <u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u><b>Phase 1 - Generating Road Masks.</b> These are the masks of the initial output from machine learning. These are not processed to a vector format. We have offered to share our
<a href="https://wiki.openstreetmap.org/wiki/AI-Assisted_Road_Tracing" target="_blank">tile service</a> to anyone.
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16685603_1205005339598346_4564649276063350784_n.jpg?oh=b2619991eef35541c7a0fb1fd7d3e19a&oe=59276BD6" target="_blank">
Sample 1</a> | <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17351717_1626376154057536_1688258658015641600_n.jpg?oh=ac96567463786eb6047cae69fb9a6c19&oe=59277824" target="_blank">
Sample 2</a> | <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/16686305_1494502210621428_618761533014409216_n.jpg?oh=3316b9f6798e3367c678fb8b93cb808d&oe=595F35AB" target="_blank">
Sample 3</a> | <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17119819_1384749424908927_2865149829640617984_n.jpg?oh=a812255978a6b91933392451b693f5ae&oe=59255A6B" target="_blank">
Sample 4</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u><b>Phase 2 - Creating Road Vectors. </b>This is the processed road vectors merged with current OSM data locally -<b>
</b>These .osm files that have NOT been edited or validated by people and would not be uploaded as is. (Please see attached)<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in">
<u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u><b>Phase 3 - Human Validation. </b>This is the final output validated by mappers twice in order to be uploaded to OSM. While we are happy to share this please note the time taken to validate, share and wait for feedback would be counter
productive because the current OSM data may be changed during this time. This increases the possibility for conflicts. We plan to map small areas at a time to ensure we are working with current OSM data and validating conflicts quickly and smoothly as we add
our generated roads. <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17351465_1228329023883542_5416323939192274944_n.jpg?oh=d34ccfb8c092940bdadf4b322a9bb33f&oe=59720FD0" target="_blank">Sample 1</a> |
<a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17365486_1399125510131415_7405155781655396352_n.jpg?oh=786797c91760722e701eabb66ad37a65&oe=5962161B" target="_blank">
Sample 2</a> | <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17365310_1444333455611506_7249929757997924352_n.jpg?oh=a5f90f3d7f0ec4c6f5224d766da0be5c&oe=596697D0" target="_blank">
Sample 3</a> | <a href="http://scontent.xx.fbcdn.net/v/t39.10482-6/17351405_1379847552085633_543677743674425344_n.jpg?oh=6f9f2c7c4e18736c28a7da8e5f2f46ea&oe=59581CCC" target="_blank">
Sample 4</a><u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-left:.5in"> <u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText">For the attached samples, please note these are not edited and would not in any way be uploaded to OSM as is. We have also provided screenshots for Phase 1 and Phase 3 as it would appear in our internal tools with the DG imagery before
being edited.<br>
<br>
Look forward to more feedback :)<br>
<br>
Best,<br>
Drishtie Patel on behalf of the OSM at Facebook Team<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText" style="margin-bottom:12.0pt"><a href="https://www.openstreetmap.org/user/DrishT" target="_blank">OSM Profile</a>
<u></u><u></u></p>
<p class="m_7937096899910185766MsoPlainText"> <u></u><u></u></p>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Imports mailing list<br>
<a href="mailto:Imports@openstreetmap.org">Imports@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/imports" target="_blank" rel="noreferrer">https://lists.openstreetmap.<wbr>org/listinfo/imports</a><br>
<br></blockquote></div><br></div>