[Tilesathome] Queue issues

[Florian Lohoff]

On Mon, Sep 08, 2008 at 09:16:15AM +0100, Ed Loach wrote:
> Once the current queue of "MissingTileRequest" tiles (and the
> occasional missing bits of Alaska I notice in the queue) are done,
> will they ever need doing again? Or do we rely on the rerender requests
> if something changes in that tile? I was wondering whether it was an
> unusual circumstance that led to over 150,000 (when I started using
> T at H, perhaps more before then) bits of ocean needing rendering.
> 
> If it is a bit of a one-off situation, then perhaps once the queue is
> empty the full upload queue will be a fairly rare occurrence after
> that, and worrying too much about the order the jobs are served to
> clients won't be such an issue?

The history is that the t at h tiles used to be on disk as individual files
which is roughly 4096x4096x1300 files which is an really worst case
scenario for every filesystem.  This also led to long delays in the
upload queue as moving the individual files took a long time.

There were some benchmarks showing that when aggregating the z12-z17
tiles into ONE file the access could get sped up a lot.
So the tileset format was changed to one file per z12-z17 which would
need a conversion of all existing tiles.

The missingTileset is the conversion e.g. the whole of 4096x4096 z12
tilesets will be rerendered. This will probably take up to half a year
to get through while in parallel keeping existing used tilesets up to
date.

Now sometimes one or the other part of the t at h falls apart and needs
fixing. t at h is a very complex and fragile construct which needs some
more tuning to actually get the most out of all the different parts.
Fixing one bottleneck just means it moves to the next smallest part so
scalability work is needed all the time.

The high number of tiles (>150k) you have seen was only a small
fraction of the needed prio4 tilesetMissing work to be done. More will
be queued from time to time to keep the clients going so the needwork
counter will rise and fall from time to time.

In my eyes the most important thing is right now to get all things fixed
up that we get a SUSTAINED throughput and not the binary works/works not 
issues we had the last weeks. Get it going reliably and long term and
then optimize our way through.

And you are right - once the million prio4 requests are through the
order is not that important anymore and can mostly be handled by the 
already implemented priority system.

Flo
-- 
Florian Lohoff                  flo at rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20080908/d0853dbe/attachment.pgp>