[OSM-talk] Missing data in rendering database on orm

[Toby Murray]

To continue the self-replies here... I was able to use the original
changeset XML, some python and a few shell tools to craft a file that
I could open in JOSM and cause it to re-upload all the objects with no
changes except for a version number bump. This caused everything to
show up in a new minutely diff that was replicated to orm to force it
to update. Everything looks as it should now in Wichita. The changeset
was almost 700 objects so it is a noticeable difference on the map.

If someone else discovers one of their changesets affected by the same
problem I could probably be convinced to repeat the procedure :)

The problem diff contains data from February 12th starting at 02:24
and ending at 06:31 UTC. That would have been a Thursday evening in
the U.S. Changesets uploaded during this time could be affected.

Toby


On Mon, Feb 22, 2016 at 11:49 PM, Toby Murray <toby.murray at gmail.com> wrote:
> After talking a little bit about this on IRC with Tom and Sarah, it
> appears this is related to a replication glitch that happened on
> February 12th. It was mentioned on the developer mailing list:
> https://lists.openstreetmap.org/pipermail/dev/2016-February/029087.html
>
> The "minutely" changeset being talked about there actually contains
> about 4 hours worth of changes due to some server problems that were
> happening at the time. So anything that was uploaded during those 4
> hours may be affected. I'm guessing the intersection of people
> uploading at that time, people who land on orm instead of yevaud for
> rendering and people who have actually noticed a glitch like this is
> pretty small. (just me so far) But this problem could affect up to
> about 340,000 objects in that diff. If objects are edited again, they
> get fixed. Although if a way is edited that contains nodes that were
> created during the 4 hour problem period there will still be problems
> with the way until all of its child nodes are edited as well.
>
> I was able to fix the gap I mentioned in my previous email by
> uploading a diff that did nothing but bump the version number of the
> way and all of its nodes.
>
> To fix everything at once would probably take either re-applying the
> last 2 weeks of diffs to orm or doing a full reimport of the rendering
> database and I'm guessing that unless some other very visible problems
> are discovered, that won't happen. And that is probably fine. This is
> just the rendering after all. The actual OSM data is ok and over time
> the number of errors will tend towards zero as things get edited. But
> it is something to be aware of if you see something odd in the tiles.
>
> I just hope this hasn't affected other consumers of minutely diffs. If
> Roland hadn't discovered it quickly in the overpass database, it could
> have led to some big problems.
>
> Toby
>
> On Sun, Feb 21, 2016 at 3:09 PM, Toby Murray <toby.murray at gmail.com> wrote:
>> I noticed something weird in the tile rendering on osm.org a couple of
>> days ago and after looking at several things it seems that one of the
>> rendering servers (orm) is missing data. I'm wondering if I'm the only
>> one to see this or if anyone else has noticed something similar.
>>
>> What I initially noticed was a gap in a road that should not have been
>> there. It can be seen in this tile:
>> http://orm.openstreetmap.org/16/15058/25351.png
>>
>> The data in the database was never in a state with a gap. I split the
>> road to tag lane count. In the same upload the way coming in from the
>> west was shortened and a new way was created. But on orm, only the
>> shortening seems to have registered and the new way creation has
>> apparently been dropped. Compare this tile to the same tile on yevaud:
>> http://yevaud.openstreetmap.org/16/15058/25351.png
>>
>> You will see not only that the gap is filled in but yevaud also has
>> several other features that are missing on orm.
>>
>> This is not a stale tile being served up from cache. These changes
>> were made on the 16th and the tile has been rendered at least 3 times
>> since then with the most recent being earlier today (on the 21st)
>>
>> The area this happened in is Wichita, KS:
>> https://www.openstreetmap.org/#map=16/37.6864/-97.2828
>>
>> But you will only see the gap if you get routed to orm via the OSM DNS
>> setup. See http://render.openstreetmap.org/ if you don't know which
>> one you are being routed to (it will say either orm or yevaud in the
>> first line)
>>
>> There is another gap if you pan west about two miles to the
>> interstate. Here I split the way to add a maxheight tag. Other things
>> I have uploaded since the 16th are showing up. It seems to be just
>> that one changeset that got partially applied to the rendering
>> database.
>>
>> Am I just really (un)lucky or has anyone else seen something similar?
>>
>> Toby