[OSM-dev] Hourly diffs are missing edits (too)

[Tom Hughes]

On 07/10/09 11:22, Frederik Ramm wrote:

> Tom Hughes wrote:
>>> Could Postgres be persuaded to abort any transaction that runs longer
>>> than "n" minutes (e.g. 30), and the we run the hourlies at hh:31 or so?
>>> That would probably be a slight inconvenience to those who happen to
>>> start 35-minute transactions but they should just learn to do their bulk
>>> imports properly ;-)
>>
>> Um... Are you being serious?
>
> Yes. In most cases, whatever HTTP client they were using to upload will
> already have terminated the TCP connection and complained about a
> timeout anyway, leading them to upload the same thing again...

Well in that case we might as well just reduce the maximum size of a 
changset surely? It will have much the same effect... We certainly could 
add a timeout though - most methods are already subject to a five minute 
timeout but it looks like none of the changeset methods are.

> (Incidentally, what happens to a changeset in these cases? Normally a
> changeset would self-close after 1hr idle time. Suppose someone starts a
> transaction on a changeset that lasts 90 minutes. If I retrieve the
> changeset after 61 minutes, will the API tell me that the changeset is
> still open but has had no edits in the last 61 minutes - or will the API
> tell me that the changeset is closed and contains no edits, and when I
> re-check an hour later, the changeset suddenly has 5000 edits?)

Yes basically. Changesets don't really auto-close at all - there is no 
cronjob that goes round closing them. Rather they have an expiry time 
which is updated when something is added to them.

So until the transaction ends you will see the pre-transaction view of 
the changeset with the old, unchanged, expiry time.

> So I would like to have *some* certainty about the run time of
> transactions. Even if it is 24 hours or so - just some value where I
> know that no transaction can possibly exceed this duration. And if we
> should find that only one promille of all transactions takes longer than
> thrity minutes then I'd be prepared to sacrifice that one transaction if
> that buys me accurate hourly diffs at 31 minutes past the hour.

Well we can't really limit at the transaction level, but we can limit 
api calls which has broadly the same effect. Long write transactions 
that we are concerned about all come from the api anyway.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/