[OSM-dev] Minute Diffs Broken
brett at bretth.com
Mon May 4 23:32:44 BST 2009
Unfortunately the minute diffs appear to be regularly missing data. In
the last 8 hours at least 3 changesets have been missed. The ones I've
noticed are 1076325, 1076998, 1077469. These have been detected by
comparing the normal minute diffs against another minute diff process
running half an hour later. I don't know what is causing these
changesets to be applied to the database so slowly, whether it's just
their size or some other factor I don't know. I don't know if this is
something that can be fixed, or whether the current osmosis extraction
method is too time-sensitive and simply broken.
At some stage over the next day or so I'll try to publish the audit
results automatically so that the problems are at least visible.
The hourly and daily diffs should be more reliable because they run with
a 30 and 40 minute delay respectively although theoretically there's no
guarantee that they're correct either.
So, any suggestions on how to fix this?
I've been trying to avoid requiring any changes to the main database in
order to keep things simple but perhaps it's unavoidable. One way
around the problem would be to introduce delta table(s) in the main
database populated by triggers on the existing history tables and
containing the ids and timestamps of changes. Osmosis could read those
tables and delete records as it processes them. It's a major change though.
This isn't an ideal forum for coming up with solutions, but I thought it
was important to ensure people are aware of the problem. I'll try to
spend some time on IRC over the next few days. Whatever the solution, I
won't have the time (or skills) to do it on my own.
More information about the dev