[OSM-dev] Minute Diffs Broken

[Matt Amos]

On Tue, Oct 6, 2009 at 3:03 PM, Florian Lohoff <flo at rfc822.org> wrote:
>
> Hi,
>
> On Tue, May 05, 2009 at 08:32:44AM +1000, Brett Henderson wrote:
>> Hi Everybody,
>>
>> Unfortunately the minute diffs appear to be regularly missing data.  In
>> the last 8 hours at least 3 changesets have been missed.  The ones I've
>> noticed are 1076325, 1076998, 1077469.  These have been detected by
>> comparing the normal minute diffs against another minute diff process
>> running half an hour later.  I don't know what is causing these
>> changesets to be applied to the database so slowly, whether it's just
>> their size or some other factor I don't know.  I don't know if this is
>> something that can be fixed, or whether the current osmosis extraction
>> method is too time-sensitive and simply broken.
>>
>> At some stage over the next day or so I'll try to publish the audit
>> results automatically so that the problems are at least visible.
>>
>> The hourly and daily diffs should be more reliable because they run with
>> a 30 and 40 minute delay respectively although theoretically there's no
>> guarantee that they're correct either.
>>
>> So, any suggestions on how to fix this?
>>
>> I've been trying to avoid requiring any changes to the main database in
>> order to keep things simple but perhaps it's unavoidable.  One way
>> around the problem would be to introduce delta table(s) in the main
>> database populated by triggers on the existing history tables and
>> containing the ids and timestamps of changes.  Osmosis could read those
>> tables and delete records as it processes them.  It's a major change though.
>>
>> This isn't an ideal forum for coming up with solutions, but I thought it
>> was important to ensure people are aware of the problem.  I'll try to
>> spend some time on IRC over the next few days.  Whatever the solution, I
>> won't have the time (or skills) to do it on my own.
>
> Is this fixed or at least somebody has an idea how to fix this? Its
> been 6 Months since i switched to hourly updates which did not suffer
> from this problem although i'd like my applications to be more current.

we hope so. there are (currently experimental) "replication" diffs
here http://planet.openstreetmap.org/minute-replicate/ in a three-deep
numeric directory structure. these diffs are produced from the
transactions in the database, so each doesn't strictly cover a minute
of edits, but they are produced with almost no delay and shouldn't be
missing any elements.

cheers,

matt