<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10pt"><div style="" class=""><span style="" class="">Roland</span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">this is awesome. Your presentation at SOTM was quite interesting and I was waiting for the revision of the database.</span></div><div class="" style="color: rgb(0, 0, 0);
font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">Below is an example where I extract places for Sierra Leone, with diff from 15-03 to 02-07. This work nicely.<br style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0);
font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">[adiff:"2014-03-15T00:00:00Z","2014-07-02T24:00:00Z"];((area["name"="Sierra Leone"];);node["place"](area);>;);out meta geom; </span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">This worked nicely. Tried to go further, extracting all diff nodes between
03-15 and 02-07 for Sierra Leone.<br style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">This worked well exporting as Raw
Data from Overpass API (116 meg output), but I receive this Ajax error message trying to run in the browser, probably because
of the size of the output: <br style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">Ajax Error - An error occured during the execution of the overpass query!<br style="" class="">Request rejected. (e.g. server not found, request blocked by browser addon, request redirected, internal server errors, etc.)<br style="" class="">Error while parsing the data (parsererror).</span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px;
font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class="">[adiff:"2014-03-15T00:00:00Z","2014-07-02T24:00:00Z"];((area["name"="Sierra Leone"];);node(area);>;);out meta geom;<br style="" class=""></span></div><br style="" class=""><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;">My next step will be to learn how to extract simultaneously various objects (ie. node, way, relation).</div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><br></div><div class="" style="color:
rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;">Thanks again for this awesome work.<br style="" class=""><span style="" class=""></span></div><div class="" style="color: rgb(0, 0, 0); font-size: 13.3333px; font-family: HelveticaNeue-Light,Helvetica Neue Light,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif; background-color: transparent; font-style: normal;"><span style="" class=""><br style="" class=""></span></div><div style="" class=""> </div><div style="" class=""><span class="" style="font-style:italic;color:rgb(0, 0, 191);font-weight:bold;">Pierre <br style="" class=""></span><br style="" class=""></div> <div class="" style="font-family: HelveticaNeue-Light, Helvetica Neue Light, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 10pt;"> <div class=""
style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div style="" class="" dir="ltr"> <hr style="" class="" size="1"> <font style="" class="" face="Arial" size="2"> <b style="" class=""><span class="" style="font-weight:bold;">De :</span></b> Roland Olbricht <roland.olbricht@gmx.de><br style="" class=""> <b style="" class=""><span class="" style="font-weight: bold;">À :</span></b> osm-talk <talk@openstreetmap.org> <br style="" class=""> <b style="" class=""><span class="" style="font-weight: bold;">Envoyé le :</span></b> Samedi 2 août 2014 2h27<br style="" class=""> <b style="" class=""><span class="" style="font-weight: bold;">Objet :</span></b> [OSM-talk] Attic data on Overpass API<br style="" class=""> </font> </div> <div style="" class=""><br style="" class="">Dear all,<br style="" class=""><br style="" class="">the attic feature of Overpass API should now
work properly.<br style="" class=""><br style="" class="">An example how to use this feature is<br style="" class=""><a style="" class="" href="http://www.openstreetmap.org/user/ikonor/diary/23329" target="_blank">http://www.openstreetmap.org/user/ikonor/diary/23329</a><br style="" class="">A big thank you to Ikonor at this point.<br style="" class=""><br style="" class="">In detail, the database has been rebuilt. I tried to do as much checks <br style="" class="">as possible, and this has shown further bugs, which in turn led me to <br style="" class="">fix this in the code and then re-build the database again. A thank you <br style="" class="">to Stephan and Markus who managed to obtain a temporary powerful server; <br style="" class="">this ultimately enabled to do the last database rebuild within less than <br style="" class="">a week.<br style="" class=""><br style="" class="">I will give details about the kind of bugs that kept me busy:<br
style="" class=""><br style="" class="">To keep the database as small as possible (currently more than 400 GB <br style="" class="">are painful, compare this to 25 GB of a PBF planet file), we store attic <br style="" class="">data as delta to the next newer version. Unchanged details like not <br style="" class="">changed tags aren't stored at all. For example, the tags of<br style="" class=""><br style="" class=""><node version="3" lat="50" lon="10" timestamp="2011-01-01"><br style="" class=""> <tag k="name" v="something"/><br style="" class=""> <tag k="highway" v="bus_stop"/><br style="" class=""> <tag k="bus" v="yes"/><br style="" class=""> <tag k="FIXME" v="check_name"/><br style="" class=""></node><br style="" class=""><br style="" class=""><node version="4" lat="50" lon="10" timestamp="2012-01-01"><br style="" class=""> <tag k="name" v="something else"/><br
style="" class=""> <tag k="highway" v="bus_stop"/><br style="" class=""> <tag k="bus" v="yes"/><br style="" class=""> <tag k="shelter" v="yes"/><br style="" class=""></node><br style="" class=""><br style="" class="">are stored as<br style="" class="">current: "name" = "something else"<br style="" class="">current: "highway" = "bus_stop"<br style="" class="">current: "bus" = "yes"<br style="" class="">current: "shelter" = "yes"<br style="" class="">attic: before 2012: "name" = "something"<br style="" class="">attic: before 2012: "FIXME" = "check_name"<br style="" class="">attic: before 2012: "shelter" = void<br style="" class=""><br style="" class="">This allows us to save space for the repeated tags "highway" and "bus".<br style="" class=""><br style="" class="">The main traces of the bug were checksum disparities in five of the 6000 <br style="" class="">checked first augmented diffs, spanning roughly 12
September to 16 <br style="" class="">September 2012. In detail, the diffs generated from the database state <br style="" class="">as of 1st October 2012 (thus, representing deltas to the then-current <br style="" class="">state of 1st October 2012) were not consistent with the diffs generated <br style="" class="">for the same minutes based on deltas to the database state as of June <br style="" class="">2014. A detailed re-generation of the augmented diffs of both database <br style="" class="">states has shown that an extra tag was present on the node 1700083447 in <br style="" class="">its attic state of September 2012 computed from June 2014 in comparison <br style="" class="">to the same node at the same attic state computed from the database as <br style="" class="">of October 2012.<br style="" class=""><br style="" class="">Do you have guessed what has gone wrong? Me not so far, so I had to <br style="" class="">understand the subtle details.
There are millions of nodes carrying <br style="" class="">tags, so what is so special about this one?<br style="" class=""><br style="" class="">It turned out that the node has moved forth and back over a significant <br style="" class="">distance, see versions 11, 14, and 15. While the movement itself is not <br style="" class="">so large (about 2 km), it happened to change its quadtile index to a new <br style="" class="">value and then back to the old value. And in version 13, after the move <br style="" class="">forth, the tag is_capital=country has been added and not changed when <br style="" class="">the node moved back.<br style="" class=""><br style="" class="">I managed in the delta computation to set a marker on the quadtile index <br style="" class="">of the older position that the tag is present, but I forgot to set a <br style="" class="">marker on the new position that the tag isn't present on earlier <br style="" class="">versions on
this quadtile index.<br style="" class=""><br style="" class="">Now: Why didn't this pop up earlier? Haven't there been tests? There <br style="" class="">have been tests, but obviously not enough, and you always only know <br style="" class="">afterwards which tests have been missing. The whole problem doesn't <br style="" class="">appear to a node when the node never existed at this place before (only <br style="" class="">3 in a million of nodes ever get back to a quadtile index where they <br style="" class="">have been before). And it doesn't appear either when the tag had any <br style="" class="">value on this older node versions (so the total number of affected nodes <br style="" class="">is less than 100), because there is then a marker for this older version <br style="" class="">and this specific key. Both cases have been tested individually, but not <br style="" class="">both together.<br style="" class=""><br style="" class="">In total:
I've worked to get the number of bugs down, and I'm confident <br style="" class="">to call the database state now consistent, but there might be other <br style="" class="">arcane bugs that affect only few objects in specific versions. So please <br style="" class="">be bold to report suspect query results to me, but be also bold in using <br style="" class="">the new attic database.<br style="" class=""><br style="" class="">Cheers,<br style="" class=""><br style="" class="">Roland<br style="" class=""><br style="" class=""><br style="" class="">_______________________________________________<br style="" class="">talk mailing list<br style="" class=""><a style="" class="" ymailto="mailto:talk@openstreetmap.org" href="mailto:talk@openstreetmap.org">talk@openstreetmap.org</a><br style="" class=""><a style="" class="" href="https://lists.openstreetmap.org/listinfo/talk" target="_blank">https://lists.openstreetmap.org/listinfo/talk</a><br style=""
class=""><br style="" class=""><br style="" class=""></div> </div> </div> </div></body></html>