[OSM-dev] Minutely changeset feed

[Toby Murray]

On Wed, Nov 7, 2012 at 10:15 AM, Matt Amos <zerebubuth at gmail.com> wrote:
> On Fri, Nov 2, 2012 at 6:34 AM, Toby Murray <toby.murray at gmail.com> wrote:
>>
>> It looks like changesets don't show up until they are closed. This
>> makes sense since then you don't have to worry about information
>> changing.
>
>
> hmm... this shouldn't be the case - changesets should show up when they're
> opened and again when they're closed. if they're opened and closed in the
> same period, it will only show the closed state. this seems to be what
> happens with most changesets - they open and close within a few seconds.

I looked at 5 consecutive diffs just now and there was not a single
open changeset. Other random ones I've looked at in the past also
don't have any open ones. I find this highly unlikely since it is
actually kind of hard to manually close a changeset in P2. So I
suspect you have a bug. Possibly the same bug that affected the weekly
planet dump until a few weeks ago when I noticed it and Tom fixed it:

http://git.openstreetmap.org/planetdump.git/commitdiff/e406711139b9eb8db9d32fa4094325ec3841082c?hp=beb331bcdf6e5565b02fcccce61be397c150bc03

The problem was that all changesets have the closed_at field set to
one hour in the future whenever the changeset is touched. This is
stored in UTC. If one hour has passed since the last change, meaning
that closed_at is in the past, the changeset is considered closed. But
the SQL that determines the changeset's openness uses the now()
function which by default returns server local time. During daylight
savings time in London this means that, the "closed_at > now()" check
always returned false so all changesets were marked as being closed in
the weekly dump.

But then again, DST is no longer in effect... So it may be something else.


>>
>> However, what will it do in the edge case where a changeset is closed
>> but then reopened? Not possible you say? I once had an upload take
>> over an hour to process. It did eventually succeed but the changeset
>> was marked as closed in my changeset list for several minutes before
>> the upload finished and then it was suddenly open again. I am assuming
>> in this case the changeset would show up twice in the minutely diffs
>> and cause the INSERT queries of all consumers to explode violently :)
>
>
> yeah... i don't think the code is able to handle this at the moment... maybe
> it would see the re-opening, but probably not. any specific examples of this
> would be great, btw, to help me start building up some test cases.

Ok well if the changeset normally shows up twice in the feed anyway
then tools will need to handle this so it shouldn't be a problem on
the consumer side. But yes, the code generating the diffs does need to
take this into account. I know the osmosis diffs are based on database
transactions so they pick up any row that has changed since the last
diff. Applied to changesets, this would actually produce an entry in
the stream every time a chunk was uploaded (in JOSM chunked upload
mode)

By examples, do you mean a changeset ID? Since this is all time based
that won't do much good since now the changeset will look like any
other changeset that I held open for 2 hours.

Toby