[Tilesathome] New t at h server

[spaetz]

On Wed, Nov 07, 2007 at 09:42:29PM -0500, Christopher Schmidt wrote:

> Tonight, I worked on the code with deelkar, making the tileset
> processing multithreaded, with great results: at first, the machine was
> only just keeping up with the requests coming in, hanging out around 240
> outstanding tilesets. After adding simple (stupid) multithreading
> support -- by using mkdir/rmdir for atomic locking -- the 8 processes we
> have running chewed through the queue pretty quick. (I think they went
> through the existing 240 tilesets plus all the incoming tilesets during
> that time in about 10min.)

Uhh, I was just thinking about how to make it multithreaded and now you already did it while I slept :-). I'll have to look what you guys did there and how.

spaetz

OK, here were my thoughts about parallelizing:

1) E.g. we can have separate  "incoming" directories for the regular and for the maplint layers (we know the layer on upload already). This way we can have two "comes along and unzips" processes which don't interfere at all.

2) Currently we store the uploaded file on /mnt/agami which is NFS attached. For unzipping the thing we will load it back to the server then, and push the unzipped files again over NFS to /mnt/agami. This means that we have lots of unnecessary network traffic. If we stored the zip files on a local directory we could avoid pushing each uploaded zip file twice through NFS. As the upload queue is around 200-400MB when full this should be doable on the local disk.

3) We could split e.g. the maplint layer and the regular tile layer on two different mounts (agami and optistor), which would distribute the IO aver disks. Combined with parallelized processing that would increase speed further.

This are relatively low hanging fruit which I can think of. If we wanted to to parallelize in a smarter way, we'd have to think more about how to achieve that.

Another option would be to e.g. keep maplint processing on the "old" dev server and just push the uploaded maplint zips over to dev and let it handle those. This would effectively create a distributed per-layer server.