[Tilesathome] My quad-core tiles at home setup

[Ævar Arnfjörð Bjarmason]

I thought I'd share with the list my t at h setup which I'm quite happy
with, it's mostly based on steve's setup[1] with some auxiliary
scripts. It's working for me on a quad-core machine so it might work
for someone else:

This is what my ~tah/ directory looks like:

avar at t:~/src/tah$ ls -l . work/
.:
total 3004
drwxr-xr-x 3 avar avar    4096 2008-08-22 15:05 script
drwxr-xr-x 8 avar avar    4096 2008-08-22 15:24 tah-1
drwxr-xr-x 8 avar avar    4096 2008-08-22 15:34 tah-2
drwxr-xr-x 7 avar avar    4096 2008-08-22 15:24 tah-3
drwxr-xr-x 7 avar avar    4096 2008-08-22 15:24 tah-4
drwxr-xr-x 8 avar avar    4096 2008-08-22 15:25 tah-upload
drwxr-xr-x 2 avar avar 3043328 2008-08-22 15:34 uploadable
drwxr-xr-x 8 avar avar    4096 2008-08-22 02:42 work

The tah-{?,upload} directories are SVN checkouts of the t at h repo[2].
The tah-[1-4] directories house clients which all upload to the
`uploadable' directory which tah-upload then picks up. This is the
relevant tilesAtHome.conf for all of them:

avar at t:~/src/tah$ head -n3 tah-{?,upload}/tilesAtHome.conf
==> tah-1/tilesAtHome.conf <==
WorkingDirectory = /home/avar/src/tah/work/tah-1
UploadToDirectory = 1
UploadTargetDirectory = /home/avar/src/tah/uploadable

==> tah-2/tilesAtHome.conf <==
WorkingDirectory = /home/avar/src/tah/work/tah-2
UploadToDirectory = 1
UploadTargetDirectory = /home/avar/src/tah/uploadable

==> tah-3/tilesAtHome.conf <==
WorkingDirectory = /home/avar/src/tah/work/tah-3
UploadToDirectory = 1
UploadTargetDirectory = /home/avar/src/tah/uploadable

==> tah-4/tilesAtHome.conf <==
WorkingDirectory = /home/avar/src/tah/work/tah-4
UploadToDirectory = 1
UploadTargetDirectory = /home/avar/src/tah/uploadable

==> tah-upload/tilesAtHome.conf <==
WorkingDirectory = /home/avar/src/tah/work/tah-upload
UploadToDirectory = 0
ForkForUpload = 0

The tah-upload client then has its uploadable directory symlinked to
../uploadable. I guess I could make the tah-[1-4] clients upload
directly to tah-upload's work directory instead but I like to have the
uploads in the main dir:

avar at t:~/src/tah$ file work/tah-upload/uploadable
work/tah-upload/uploadable: symbolic link to
`/home/avar/src/tah/tah-upload/../uploadable'

To run it all I open several windows in GNU screen, one of them is the
upload client which always runs in an upload loop:

avar at t:~/src/tah/tah-upload$ perl tilesGen.pl upload_loop
- Using working directory /home/avar/src/tah/work/tah-upload
- Keeping ZIP files after upload
This is version 10049 (Rapperswil) of tilesgen running on linux, ID: 84105960
[#0   0% uploadloop] waiting for new ZIP files to upload   0:00...

In the rest of the tah-[1-4] directories I'm running t at h jobs to
reneder areas I care about, usually something I've just mapped. Most
of the time I only have 1-2 out of the 4 possible clients active.

To render the area I've just mapped I download the area from the API
I've been mapping, aggregate all its coordinates and use a custom
script (osm-to-tilenumbers.pl) to get a list of all the applicable
tiles that have changed in say the last 6 hours. I then run
tilesGen.pl through a monitoring script which takes care of rendering
the list of tiles given on STDIN and restarting (with appropriate
calls to sleep()) tilesGen.pl if it were to exit uncleanly for some
reason, usually due to the t at h api being down or network troubles:

avar at t:~/src/tah/tah-1$ time (cat /tmp/rvk-map.osm | perl
../script/osm-to-tilenumbers.pl --zoom 12 --eval 'abs($sec_ago) <
(60**2 * 6)' | perl ../script/tilesGen-monitored.pl --zoom 12)

I maintain both those scripts in a git repository[3] if anyone's
interested in them I could maintain one or both of them in the
tilesAtHome svn directory (although I currently don't have commit
access).

1. http://lists.openstreetmap.org/pipermail/tilesathome/2007-December/001678.html
2. http://svn.openstreetmap.org/applications/rendering/tilesAtHome
3. http://git.nix.is/?p=avar/tah-avar;a=tree;h=master;hb=master