[Tilesathome] Queue issues

[spaetz]

On Mon, Sep 08, 2008 at 01:01:43AM +0100, Ed Loach wrote:
> 
> Could perhaps the server stop giving out new requests when active requests + number of requests in update queue > ~95% of maximum upload queue? That way the upload queue is unlikely to fill completely, which seems to be where the problems are at the moment (I've been trying to upload the same result for about 8 hours now, and the intervals between retries are somewhere about 600 seconds). 

There's hundreds of active jobs out there are any one time (if the server is not broken again :-)). And responses come back sometimes many hours after a request has gone out. So stopping the handing out of requests would take a bit of time until the system reacts, and I remember horible stuff back from my physics classes, with resonance requencies, self reinforcing circuits, leading to eventualy breakdowns. I'd rather have a smooth flow than trying to start and stop things in this system.

I don't see a full upload queue as a particular problem, it's just another bottleneck that needs elimination -- unless of course, it blocks high priority requests from being uploaded because the server is busy with empty sea tiles --.

I prepared the server to send out the priorities of a request to the client, it just needs a small change and the client will know about it.
Clients can the decide, for example to upload priority 1 requests until the upload queue is full. They could hold back priority 4 requests when the upload queue is >50% full (or something similar). This way we never block higher priority requests