[Tilesathome] Rate Limiting & be gentle with you requests

[Raphael Studer]

> Hi all,
> up to yesterday we would accept all requests and hand them out to as many renderer clients as were willing to take up jobs. As the 2500 concurrent render jobs were too much for the API server TomH had to restrict access from renderer clients yesterday in order to keep the API server alive. So changes are needed!
>
> I implemented a simple rate limiting, which basically only hands up to 500 active jobs out at the same time. While that number might need tweaking, it seems to work reasonably well (although the API seems still too slow, we currently have very little uploads).
>
> On the other hand, making requests is not limited at all. Yesterday LosHawlos requested at least 2.5k tilesets and over night within a couple of hours somebody fired off >2.5k requests with a bulk rerendering of the whole of norway, leading to around 5000 bulk requests within 24 hours. This leads to huge pending queues, which we will probably not be able to process within a reasonable time frame. So I am thinking of limiting the number of requests to something like 10 concurrent priority 1 requests and 50 concurrent priority 2 requests for any given username. As the username is an arbitrary string which can be supplied by the requester, this won't help against malicious users, they could simply pick a random string for each request, but it would help to restrict bulk requests if we reply with a "access denied" or "server busy" message if you have too many active requests going on.
>
> What do people think? Should we limit request making and in which ways would you think it should be done?

Instead of limiting the requests per user, it would be possible to
lower the priority of requests. Lets say first priority for at least
10 requests per user, second priority for the next 100 requests and
third priorit for further requests from this user.
To stop malicious using of the requests, it would also be possible to
choose the IP as criteria for choose a priority instead of the
username.

An other limiting thing could be, remove all not active requests older
than 48 hours (if this isn't done allready).

Raphael