[OSM-dev] Minutely changeset feed

[Matt Amos]

On Wed, Nov 7, 2012 at 5:22 PM, Toby Murray <toby.murray at gmail.com> wrote:

> On Wed, Nov 7, 2012 at 10:15 AM, Matt Amos <zerebubuth at gmail.com> wrote:
> > On Fri, Nov 2, 2012 at 6:34 AM, Toby Murray <toby.murray at gmail.com>
> wrote:
> >>
> >> It looks like changesets don't show up until they are closed. This
> >> makes sense since then you don't have to worry about information
> >> changing.
> >
> >
> > hmm... this shouldn't be the case - changesets should show up when
> they're
> > opened and again when they're closed. if they're opened and closed in the
> > same period, it will only show the closed state. this seems to be what
> > happens with most changesets - they open and close within a few seconds.
>
> I looked at 5 consecutive diffs just now and there was not a single
> open changeset. Other random ones I've looked at in the past also
> don't have any open ones. I find this highly unlikely since it is
> actually kind of hard to manually close a changeset in P2. So I
> suspect you have a bug. Possibly the same bug that affected the weekly
> planet dump until a few weeks ago when I noticed it and Tom fixed it:
>

ah... just noticed a massive bug - i was assigning both created_at and
closed_at from the database's created_at! no wonder it thought everything
was already closed. right, i'll go fix that.


>
> http://git.openstreetmap.org/planetdump.git/commitdiff/e406711139b9eb8db9d32fa4094325ec3841082c?hp=beb331bcdf6e5565b02fcccce61be397c150bc03
>
> The problem was that all changesets have the closed_at field set to
> one hour in the future whenever the changeset is touched. This is
> stored in UTC. If one hour has passed since the last change, meaning
> that closed_at is in the past, the changeset is considered closed. But
> the SQL that determines the changeset's openness uses the now()
> function which by default returns server local time. During daylight
> savings time in London this means that, the "closed_at > now()" check
> always returned false so all changesets were marked as being closed in
> the weekly dump.
>
> But then again, DST is no longer in effect... So it may be something else.


yeah, i thought i handled that - all calls in the code are UTC, although i
see i've used now() in one SQL statement, but it's just to get a starting
list and is filtered against the correct time later in the code.

i will have to do some investigation...


> >>
> >> However, what will it do in the edge case where a changeset is closed
> >> but then reopened? Not possible you say? I once had an upload take
> >> over an hour to process. It did eventually succeed but the changeset
> >> was marked as closed in my changeset list for several minutes before
> >> the upload finished and then it was suddenly open again. I am assuming
> >> in this case the changeset would show up twice in the minutely diffs
> >> and cause the INSERT queries of all consumers to explode violently :)
> >
> >
> > yeah... i don't think the code is able to handle this at the moment...
> maybe
> > it would see the re-opening, but probably not. any specific examples of
> this
> > would be great, btw, to help me start building up some test cases.
>
> Ok well if the changeset normally shows up twice in the feed anyway
> then tools will need to handle this so it shouldn't be a problem on
> the consumer side. But yes, the code generating the diffs does need to
> take this into account. I know the osmosis diffs are based on database
> transactions so they pick up any row that has changed since the last
> diff. Applied to changesets, this would actually produce an entry in
> the stream every time a chunk was uploaded (in JOSM chunked upload
> mode)
>

yeah, i wanted to avoid the changeset being in the feed too many times
(e.g: chunked upload). ideally only when it is opened and closed, although
there are situations where it probably prints out more often.


> By examples, do you mean a changeset ID? Since this is all time based
> that won't do much good since now the changeset will look like any
> other changeset that I held open for 2 hours.


i'll have to find more inventive ways to test the code, then ;-)

cheers,

matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20121107/272ecbf9/attachment-0001.html>