[OSM-dev] Minute Diffs Broken

[Brett Henderson]

Hi Everybody,

Unfortunately the minute diffs appear to be regularly missing data.  In 
the last 8 hours at least 3 changesets have been missed.  The ones I've 
noticed are 1076325, 1076998, 1077469.  These have been detected by 
comparing the normal minute diffs against another minute diff process 
running half an hour later.  I don't know what is causing these 
changesets to be applied to the database so slowly, whether it's just 
their size or some other factor I don't know.  I don't know if this is 
something that can be fixed, or whether the current osmosis extraction 
method is too time-sensitive and simply broken.

At some stage over the next day or so I'll try to publish the audit 
results automatically so that the problems are at least visible.

The hourly and daily diffs should be more reliable because they run with 
a 30 and 40 minute delay respectively although theoretically there's no 
guarantee that they're correct either.

So, any suggestions on how to fix this?

I've been trying to avoid requiring any changes to the main database in 
order to keep things simple but perhaps it's unavoidable.  One way 
around the problem would be to introduce delta table(s) in the main 
database populated by triggers on the existing history tables and 
containing the ids and timestamps of changes.  Osmosis could read those 
tables and delete records as it processes them.  It's a major change though.

This isn't an ideal forum for coming up with solutions, but I thought it 
was important to ensure people are aware of the problem.  I'll try to 
spend some time on IRC over the next few days.  Whatever the solution, I 
won't have the time (or skills) to do it on my own.

Brett