[OSM-dev] [OSM-talk] Handling of towns with different or alternative names

[Dave Stubbs]

2009/1/28 Richard Fairhurst <richard at systemed.net>:
>
> Stefan de Konink wrote:
>> If you can agree that this tag/nds out of order thing could be the
>> main reason for these strange inserts, I am happy to help you
>> search for the source that reflects this.
>
> I have an open mind as to what it might be.
>
>> For some reason I think that the actual API call to update a way
>> (hence: new timestamp/user) *is* done first from a code
>> perspective, but is not finished when the request is done for
>> current tables vs history tables. If you say 'but i create an xml
>> and PUT it too' /way/<id> like anything else, then that is very
>> interesting situation and you maybe right we are looking for the
>> wrong editor.
>
> For its transport format, Potlatch uses AMF rather than XML. Speed is vital
> in an online editor, like Potlatch (or, indeed, the original applet). When
> using Flash, AMF both saves bandwidth and serialisation/deserialisation
> time.
> http://www.jamesward.com/blog/2007/04/30/ajax-and-flex-data-loading-benchmarks/
> will give you some benchmarks. (Plus, of course, it doesn't suffer from the
> memory leaks that we seem to have with Ruby's XML handling.)
>
> amf_controller.rb then takes these AMF messages, and calls exactly the same
> Rails methods as the XML API does.
>
> I would strongly recommend you look at the code that saves ways:
>
> http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb#L330
>
> FWIW, my experience is that data inconsistencies of this sort happen mostly
> when the server is under very heavy load. If a process is killed halfway
> through a write operation, then obviously you're going to get some sort of
> inconsistency. You can order the operations so that this is less likely (cf
> http://trac.openstreetmap.org/changeset/13184/), but as has been said here
> extensively, this kind of stuff is always going to happen without
> transactions.
>


I've just done some research on this.

Basically, the short version is that OldWay.save_with_dependencies!
which does the save of the history has some hacks to get round rails
being a bit shit if you don't have a unique column as your index. This
means that you can't save two or more versions of a way with the same
timestamp (which only has 1 second resolution); if you do then it
results in duplicate tags on the first version, and an empty entry for
the second version. The second update will throw an error as the
way_node primary key constraint on is violated (i've tested with
Potlatch on a dev env and the error icon does popup), but by that
point the tags are already written.

This all matches the history output -- however the current way output
shouldn't be affected afaict.
Stefan: before you correct these, is the data as returned by
http://www.openstreetmap.org/api/0.5/way/nnnn correct? I know the
/history is broken, and therefore the osmosis generated diffs will be
broken too.

You can achieve the same results with the standard API (I have tested
and shown this to be true).
I think it's just that it's very very rare as other editors don't have
the same approach/frequency to saving ways. At least, that's my
interpretation, and explains why these things are more likely to
happen during heavy load as update requests can get queued behind map
calls, and then all get processed in the same second with the same
timestamp. And it also gets Potlatch/amf_controller off the hook, at
least directly.


To mitigate: save up writes in potlatch... try not to send any updates
with others already pending (that should happen already anyway)

To fix: OldWay needs an alternative hack -- I can't think of one right
now that doesn't involve adding another column which would use quite a
lot of space for very little reaason.


Dave