<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><title>RE: [Imports] (Estonia) Maa-amet building geometry update</title><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ET link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>I’ve ran the script in few short batches updating live data and modified about 180 first buildings. <a href="https://www.openstreetmap.org/user/fghj753_import/history">https://www.openstreetmap.org/user/fghj753_import/history</a> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>There were however some worrying log lines </span><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>'Remote end closed connection without response' which i believe were caused by too frequent NodeWays API requests (in order to prevent editing unrelated ways). I think setting 0.1 sec delay before every node query might help to mitigate the error. This would make whole script run time 5-6h longer.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Fghj753</span><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif'> fghj753 <fghj753@hot.ee> <br><b>Sent:</b> Tuesday, June 1, 2021 2:19 PM</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'><br><b>To:</b> imports@openstreetmap.org<br><b>Subject:</b> Re: [Imports] (Estonia) Maa-amet building geometry update<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>I’d like to run two changesets with ~200 buildings on live server to see how upload runs on other servers and so you could review first edits. I’ve added 60s delay after every changeset (40x) and 0.4s delay after every modified building (there are ~32k buildings), creating total delay of 04:13 h, in addition to projected 10h upload time.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Remember that these buildings are in rural area, hence the small changesets. When script reaches urban areas, changesets will be much larger.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>May I continue to upload?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> fghj753 <</span><a href="mailto:fghj753@hot.ee"><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>fghj753@hot.ee</span></a><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>> <br><b>Sent:</b> Monday, May 31, 2021 3:14 PM<br><b>To:</b> </span><a href="mailto:imports@openstreetmap.org"><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>imports@openstreetmap.org</span></a><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'><br><b>Subject:</b> Re: [Imports] (Estonia) Maa-amet building geometry update<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>></span><span lang=EN-GB> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>I really, really, really dislike such debris on objects. Especially as it is often stale anyway - people</span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>> updating building shape based on aerial/survey rarely remove such tags.</span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>> Though please ignore me if local community has a different preference. </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>> It is also a personal preference that is (I think) well motivated, not OSM rule.</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Only reply from local mailing list agrees with you. I also sent clarifying question to</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Maa-amet / Land Board, but I don’t think they will reply anytime soon.</span><span lang=EN-GB> </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>></span><span lang=EN-GB> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>For technical side: interrupt script after it made N edits, then restart it the next day?</span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>></span><span lang=EN-GB> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>Or sleep for N seconds after each edit?</span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>></span><span lang=EN-GB> </span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>></span><span lang=EN-GB> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>When I run bot edit it typically sleeps one minute after every changeset</span><o:p></o:p></p><p><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>> </span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Uploading changeset of 10000 changes took around 20 min on local VM. I’m afraid delaying just</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>1 minute between sets is</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>not enough.</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>For comparison, your bot made 100-300 changes per changeset in</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>less than 2 minutes and then waited for 1 min. Waiting 0.5-1 sec per building (not</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>node</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>change) may double the time consumed, but</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>seems to be more fair solution to delay edits. Also in that case uploading would start approaching full day, allowing to start import any time.</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'> Honestly I’d prefer to run import</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>overnight, but then leaving 2-4 hour chunk for next day doesn’t make much sense. The</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>python</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>script does support splitting</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>the upload.</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>My personal objective on the matter is to complete import by Friday morning, but I probably don’t have time on Thursday</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>evening to start it, meaning that import should be scheduled on Wednesay evening.</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>></span><span lang=EN-GB> Though I think that adding LIDAR based height data should be mentioned in summary posted here</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Not much to comment here</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>other than posting direct link to table:</span><span lang=EN-GB> </span><a href="https://wiki.openstreetmap.org/wiki/Maa-amet_building_geometry_update#Tagging_Plans"><span lang=EN-GB style='font-family:"Calibri",sans-serif'>https://wiki.openstreetmap.org/wiki/Maa-amet_building_geometry_update#Tagging_Plans</span></a><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'> and pasting table cell in question:</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>></span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Dataset also contains lidar-measured building height data for some buildings. Height is only added if any nodes of the building were modified, height=* is not</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>present yet and height recorded in dataset is at least 3 m.</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>There were concerns about StreetComplete not asking building:levels quest for buildings with height data, but Mateusz has already opened PR on SC’s repo to solve the</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>problem from</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>the other</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>end.</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Height data is stored as integer, meaning</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>1m precision. Also I’m planning to remove bit under 200 instances of</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>undocumented (CityIdx), legacy (created_by) and duplicate tags (name == addr:housenumber) on buildings, where geometry is updated too. Planning to add height data and</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>previous</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>local building import’s signature key maaamet:ETAK.</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>I’ve added two rows to</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Import/Catalogue, but</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>I also took liberty to unify</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>date formatting of these 3 tables. That</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>raised an odd question: How many days were in November 2009 (looking at</span><span lang=EN-GB> </span><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>„Spanish administrative borders“ Import)?</span><o:p></o:p></p><p><span lang=EN-GB style='font-family:"Calibri",sans-serif;color:#1F497D'>Fghj753</span><o:p></o:p></p><p><b><span lang=EN-GB style='font-family:"Calibri",sans-serif'>From:</span></b><span lang=EN-GB style='font-family:"Calibri",sans-serif'> Mateusz Konieczny via Imports <</span><a href="mailto:imports@openstreetmap.org"><span lang=EN-GB style='font-family:"Calibri",sans-serif'>imports@openstreetmap.org</span></a><span lang=EN-GB style='font-family:"Calibri",sans-serif'>><br><b>Sent:</b> Monday, May 31, 2021 11:20 AM<br><b>Cc:</b> Imports <</span><a href="mailto:imports@openstreetmap.org"><span lang=EN-GB style='font-family:"Calibri",sans-serif'>imports@openstreetmap</span><span lang=EN-US style='font-family:"Calibri",sans-serif'>.org</span></a><span lang=EN-US style='font-family:"Calibri",sans-serif'>><br><b>Subject:</b> Re: [Imports] (Estonia) Maa-amet building geometry update</span><o:p></o:p></p><p>May 30, 2021, 23:10 by <a href="mailto:fghj753@hot.ee">fghj753@hot.ee</a>:<o:p></o:p></p><p><span lang=EN-GB>I have few finalizing questions. Should “source:*=Maa-amet 2021” tag be duplicated onto elements (buildings) and if so, under which subkey (source:geometry?), or would “source=Maa-amet 2021” on changeset suffice? Previous address imports have added source:addr to each building and also regular source tag on changeset.</span><o:p></o:p></p><p>I really, really, really dislike such debris on objects. Especially as it is often stale anyway - people<o:p></o:p></p><p>updating building shape based on aerial/survey rarely remove such tags.<o:p></o:p></p><p>Though please ignore me if local community has a different preference. <o:p></o:p></p><p>It is also a personal preference that is (I think) well motivated, not OSM rule.<o:p></o:p></p><p><span lang=EN-GB>Secondly import guidelines mentioned something about spreading import across wider timeframe, citing old Tiger imports. How important is that and how to achieve that?</span><o:p></o:p></p><p>I suspect that reason for that is that it is easier to spot runaway script damaging data?<o:p></o:p></p><p>For technical side: interrupt script after it made N edits, then restart it the next day?<o:p></o:p></p><p>Or sleep for N seconds after each edit?<o:p></o:p></p><p>When I run bot edit it typically sleeps one minute after every changeset<o:p></o:p></p><p>(that are grouped into groups of close edits).<o:p></o:p></p><p><span lang=EN-GB>Are there some recommendations on how to improve import documentation?</span><o:p></o:p></p><p><a href="https://wiki.openstreetmap.org/wiki/Maa-amet_building_geometry_update">https://wiki.openstreetmap.org/wiki/Maa-amet_building_geometry_update</a> seems OK<o:p></o:p></p><p>Though I think that adding LIDAR based height data should be mentioned in summary<o:p></o:p></p><p>posted here<o:p></o:p></p><p><span lang=EN-GB>I wasn't sure what exactly to write into import documentation as wiki documentation of all previous Estonian imports combined have roughly length of this email. Currently I haven’t added row to Import/Catalogue yet. </span><o:p></o:p></p><p><span lang=EN-GB>Should I add line for 2008 import as well?</span><o:p></o:p></p><p>Seems to be a good idea.<o:p></o:p></p></div></body></html>