[Tilesathome] Rate limiting on upload page

[David Earl]

Since you are uploading the file anyway (and that's a big chunck of the work
and itself imposes load), why not leave the file on the server along with a
little control file to say what it is (if indeed you need one - location on
disk may suffice) and then process the incoming files independently of the
uploads when the server is quiet. That is, have the incoming tiles form a
queue.

That way, the client doesn't need to know the status, just the process that
is contemplating doing something expensive.

David


> -----Original Message-----
> From: tilesathome-bounces at openstreetmap.org
> [mailto:tilesathome-bounces at openstreetmap.org]On Behalf Of OJW
> Sent: 29 May 2007 19:42
> To: tilesathome at openstreetmap.org
> Subject: [Tilesathome] Rate limiting on upload page
>
>
> We're testing a system that only accepts uploads if the server is
> more than a
> certain amount idle.  The reason being that map viewing is a
> realtime thing
> that people rely on for a certain response rate, so some
> CPU/disk-access time
> needs to be reserved for that no matter how many uploads are coming in.
>
> The message is currently "501, server is quite busy", and the limit is
> currently "must be > 20% idle".
>
> Unfortunately (because it's in PHP), this calculation is only
> done _after_ the
> file is transferred, which isn't ideal for any contributors with limited
> bandwidth.
>
> If you'd like to know in advance whether your upload is likely to
> succeed,
> I'll create a page that says "GO" or "STOP" depending on what the current
> status is. Don't forget that status may change rapidly!
>
> http://dev.openstreetmap.org/~ojw/Upload/go_nogo.php
>
> Any suggestions for limiting mechanisms that work earlier in the upload
> process (e.g. apache config) would be appreciated.  Remember that
> the system
> mustn't oscillate - anything which checks once per minute and
> allows/disallows all uploads within the subsequent minute will give you a
> cycle of one minute overloaded, next minute totally idle.
>
> Why not rate-limit requests instead?  Well there's a long and variable
> time-delay between requests and uploads, plus many people don't take any
> notice of the requests queue and run their own one.  Limiting at
> the upload
> stage is likely to be more reliable at the moment.  But if you
> have an idea
> for rate-limiting in requests queue, or in the client, or at any
> other point
> in the application, please post your code to SVN.
>
> If the tiles at home client would like to implement it, we could
> have a priority
> for uploads, so that if the request you took was priority-1 (means that
> someone is waiting for it) then you could say so while uploading and get
> better access to the upload page.
>
> Regards,
>
> OJW
>
>
> _______________________________________________
> Tilesathome mailing list
> Tilesathome at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome