[Tilesathome] Rate limiting on upload page

[Robert (Jamie) Munro]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frederik Ramm wrote:
> Hi,
> 
>> We're testing a system that only accepts uploads if the server is more than a 
>> certain amount idle. The reason being that map viewing is a realtime thing 
>> that people rely on for a certain response rate, so some CPU/disk-access time 
>> needs to be reserved for that no matter how many uploads are coming in.
> 
> I still think that we are a community of people with a common goal. Some 
> of us may be a little brighter and some a little more stupid but nobody 
> is malicious. If we are given workable mechanisms to ensure the common 
> goal is not hurt (by e.g. too many uploads) then we will use them.
> 
> I dislike the attitude that tiles at home contributors must be forced to do 
> or not do all kinds of things.
> 
> I would have thought that any development time would be better spent in 
> just giving the client a method to throttle uploads for the time being 
> than to think about how to efficiently harden the server 
> against evil (or stupid) users who disregard the "red light".

> (by using: 
> 
> http://dev.openstreetmap.org/~ojw/Upload/go_nogo.php

When was that invented? I should probably add checking the state of that
to my bulk requester. Currently it doesn't issue requests if
http://dev.openstreetmap.org/~ojw/Log/Requests/History/munin.php
says there are more than 20 already in the queue. (it only checks every
50 requests, so as to not overload the system by checking, so it could
put up to 70 in the queue).

Also, if the request takes more than 0.2 seconds to execute, it
multiplies the time it took by four before trying again. If the server
is very slow, this can delay it for more than a minute between
requesting each tile.

I could adapt it to look at that go_nogo script as well. I've put the
script in svn, so that people can comment, but it's probably better if
people suggest areas they want bulk-rerendered to me and I run it,
rather than everyone run their own copies.

Something that occurred to me is if upload.pl fails to upload, it should
retry (with exponential backoff and/or checking go_nogo), and if it
still fails the tilesGen loop should exit, rather than it carry on
downloading tiles and never uploading. Currently, it seems that if
upload.pl fails for whatever reason, tilesGen just go's around in
another loop.

Robert (Jamie) Munro
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGXLIlz+aYVHdncI0RAgJNAKD8GDTZiiOrTrlYVjnFbNHzVPlUwgCgtJ5u
fprm41AI+9qbARWZihDK9Y8=
=Oc4R
-----END PGP SIGNATURE-----