[OSM-dev] More responsive T at H performance

[Frederik Ramm]

Hi,

> the possible patch would look like this.  Any thoughts or problems
> that I didn't think of?

Your patch would work but I'd like a more thorough tuning. We're dealing 
with three separate scarce resources in t at h:

a. download bandwidth
b. CPU/disk performance
c. upload bandwidth

Currently, CPU and upload pipe are waiting while we download a job, then 
download and upload pipes are unused while we're doing the rendering, 
and then download pipe and CPU are unused while we're uploading.

Your patch increases latency a bit - some results get delivered faster - 
but it doesn not increase throughput.

It should be possible to beef up overall performance considerably if we 
ran a constant uploader in the background which would constantly check 
if there's already enough data for an upload and start uploading as soon 
as there is. The uploader would be started whenever tilesGen starts up 
and finds no uploader is running.

That way, the next job can be started immediately after completion of 
one. The tilesGen script should do an extra safety check before it 
downloads a request ("if more than 50 mb of tile data accumulated and 
not uploaded, stop because the upload server is probably broken").

The only reason I haven't built something like that yet is fear of 
breaking the Windows setup (don't know if background processes need 
special handling there).

A simple version of my idea would be just adding a "&" where upload.pl 
is called so that it works in the background. This I don't recommend 
though as things will get ugly if tilesGen gets a new tile done before 
the last tile's uploader has finished (you'll get more than one uploader 
working on the same directory).

I believe the tilesPipe at home project did have something like this 
planned but never went into production.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00.09' E008°23.33'