[OSM-dev] More responsive T at H performance
frederik at remote.org
Thu Apr 19 21:58:08 BST 2007
> the possible patch would look like this. Any thoughts or problems
> that I didn't think of?
Your patch would work but I'd like a more thorough tuning. We're dealing
with three separate scarce resources in t at h:
a. download bandwidth
b. CPU/disk performance
c. upload bandwidth
Currently, CPU and upload pipe are waiting while we download a job, then
download and upload pipes are unused while we're doing the rendering,
and then download pipe and CPU are unused while we're uploading.
Your patch increases latency a bit - some results get delivered faster -
but it doesn not increase throughput.
It should be possible to beef up overall performance considerably if we
ran a constant uploader in the background which would constantly check
if there's already enough data for an upload and start uploading as soon
as there is. The uploader would be started whenever tilesGen starts up
and finds no uploader is running.
That way, the next job can be started immediately after completion of
one. The tilesGen script should do an extra safety check before it
downloads a request ("if more than 50 mb of tile data accumulated and
not uploaded, stop because the upload server is probably broken").
The only reason I haven't built something like that yet is fear of
breaking the Windows setup (don't know if background processes need
special handling there).
A simple version of my idea would be just adding a "&" where upload.pl
is called so that it works in the background. This I don't recommend
though as things will get ugly if tilesGen gets a new tile done before
the last tile's uploader has finished (you'll get more than one uploader
working on the same directory).
I believe the tilesPipe at home project did have something like this
planned but never went into production.
Frederik Ramm ## eMail frederik at remote.org ## N49°00.09' E008°23.33'
More information about the dev