[Tilesathome] Rate Limiting & be gentle with you requests

[spaetz]

On Fri, Aug 10, 2007 at 12:26:02PM +0100, Robert (Jamie) Munro wrote:
> It's not the number of active jobs that's important. It's the number
> that are in the "requesting data from the API" stage of the process.

Sure, but I have no way of finding that out. I hand out a request for atileset to a renderer and get back some images later (or not). I cannot possibly lknow how many blank tiles are involved, nor whether they are all rerequesting data because the API was down for some time or if they are all happily rendering along. I don't even know if a renderer simply took a job and discarded it (because the corresponding computer was e.g. shut down). So I have to live and deal with some number I have, and that is the number of currently actuve jobs.

> I think we need to concentrate on directly making sure that T at H clients
> don't overload the API.

Sure, patches accepted.
 
> When T at H clients request data from the API, they need to request it
> low-priority. The API should be able to not give it to them if it is a
> bit busy.

Agreed. However that is nothing I cn control from the t at h server side. I agree, it should be tackled though.
 
> > What do people think? Should we limit request making and in which ways
> > would you think it should be done?
> 
> Requests go into a queue, and can stay there. There's no point in
> limiting them entering the queue. It might be worth giving certain
> requests in the queue a higher priority, but if you try to rate limit
> the incoming queue, I'll just have to make a local queueing system and
> have a daemon put my requests on the real queue at a slower rate, which
> is just a waste of everyone's time.

Right, but we have to time out requests at some point in time, if they aren't handled or we will simply have a render the whole world queue all the time. Or are you suggesting, requests should be resent to clients infinitely until one fulfills that request?

spaetz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20070810/014bba09/attachment.pgp>