[Tilesathome] "done" down and "active" through the roof

[Matthias Julius]

Dirk-Lüder Kreie <osm-list at deelkar.net> writes:

> Matthias Julius schrieb:
>> Hi all,
>> 
>> The "done" queue is now down to about half of what it was a few days
>> ago.  Did something happen to cause this?  Or do people now circumvent
>> the request queue and send in unsolicited uploads?
>
> the OSM api was down for several hours last night, and since "done" is a
> 48 hour average (normalized to 1 hour) it went down with it.
  ^^^^^^^^^^^^^^^
Shouldn't that be changed?  I think this number would be a lot more
useful if the averaging period wasn't that long.

>
>> At the same time "active" is at 16k and my client is idle.  There must
>> be clients out there that grab every job they can while having a huge
>> backlog.
>
> this was also partly due to the API downtime. I would estimate 10-15k
> jobs there to be "lost".
>
>> Should the server keep track of who is doing which job and not hand
>> out new ones while there are too many pending?
>
> We could use a better queue management, yes. before the server overload
> we had a system that re-requested the active jobs that hadn't been
> turned in after 6 hours. This is obviously only useful if clients can
> upload shortly after rendering.
>
> Instead of that approach I suggest we use a job-reinsertion scheme that
> moves the oldest X "active" jobs to "new" to avoid swamping the queues
                                  ^
that are at least 6 hours old

> with re-requests. X to be chosen depending on load and queue length
> (10-100 maybe?).

I think the expiry mechanism should be re-enabled at some point.  I
rather have old jobs in "new" than in "active" for days or weeks.
Especially if your estimate above of "lost" jobs is correct.

Now that the client is behaving much better and doesn't ask for a new
job until the previous ones are uploaded it should always be possible
to upload within the 6 hours time limit.

Matthias