[Tilesathome] Distributing tiles at home server chores?

[David Earl]

> Unless we manage to drastically optimise the software at this point, we
> should think about distributing "B" to more than one server. How could
> this be done?

If we're so heavily underpowered at the moment, it will only get worse as
the data massively grows and the number of users grow. A single machine will
only be a stop gap. Distribution is surely the only way to go. A growing
number of cheap machines allows us to spread load and add disk as the number
of tiles grows.

Here's my suggestion on how to do it:

- maintain a database of where each tile is. User asks for a set of tiles in
two stages: ask the database where the tiles are and then go off to get the
actual files, quite likely from different sources.

- on upload, ask the database for where to put a tile set (maybe different
places for subsets). Database server chooses server randomly, but because
the server is doing the allocation it can be tuned (e.g. don't allocate the
same server twice consecutively). On completion of uploads slave servers
register their successful uploads with the database.

While a deliberate algortihm for distributing tiles sounds appealing, so
often random allocation turns out to be just as good and is really simple.

David