[Tilesathome] Thoughts about smoothing the load on dev

[Jon Schlueter]

On 6/5/07, Martijn van Oosterhout <kleptog at gmail.com> wrote:
> On 6/5/07, Dirk-Lüder Kreie <osm-list at deelkar.net> wrote:
> > A reset to 0 should be OK, if not everyone waiting for upload resumes
> > uploading in a short timeframe, so just randomizing the upload delays a
> > bit should smooth things out already.
>
> I disagree. We have, on the whole, a load of clients trying to upload
> faster than the server can handle. To reduce the load we add delays on
> the client side. It may be that the optimal upload pattern is for
> every client to delay 8 seconds between each upload. But the system
> can never stabilise to that, because as soon as one upload goes
> through it starts uploading as fast as possible.
>
> The simple suggestion is maybe too simple, but it allows the system to
> stabilise to a delay other zero, which the current code can't do.

I think a variant on the patch I posted here might help with the clients....
A slight tweak is needed to help with timing and responsivness
and I would agree that deleting any downloads which are over X hours
hold would be in general a very good thing to do....

If we extend the failed uploads count to add in delays to the upload
and rendering request.  Time to delay might be good to delay.... Also
might be good to delay and slow down when trying to render a file and
it get's no data back... or a server too busy response while trying to
get data from the server to render....
That would be a good point to delay and scale back.

Any pointers as to where that might be in the tilesGen.pl file?

I already added a DelayOnFailure sub function to the tilesGen.pl
Now all that has to be done is figure out where to hook in to
increment failure count on failed download of a tile as well....

and to tweak the scaleback of failure number when upload completes on time....