[OSM-dev] Tiles at home - current disk use

[OJW]

On Friday 18 May 2007 13:09, SteveC wrote:
> but the new drive isn't actualy being used

OK, let's explain what's happening with disks:

Background info 1: "disk1" filled up, and write performance was adversely 
affected by using a disk at near its capacity limit.

Background info 2: Early versions of land/sea rendering created a lot of blank 
blue tiles which were stored as images.  We'd prefer to store blank tiles as 
a database entry (or binary file, etc), but detecting and removing the old 
ones takes a lot of CPU and disk access on the server

Background info 3: The new land/sea rendering processes means that it's 
desirable to have re-renders of everything, so that coastlines will show up 
on the map. 

So...

New uploads are all being saved to "disk2".  Disk2 is amost empty, and has 
more than enough space to store all the tiles that rendering the current OSM 
dataset will generate.  Eventually this scheme will need to be replaced with 
one which distributes data across both disks, but we have some time to think 
about that rather than doing it now without prior testing.

I've asked a couple of people to try requesting re-renders of everything.  
Basically, look for tiles generated before 2007-05-19 and request them.

mod_rewrite looks on disk2.  Tiles on disk2 will therefore be faster to 
access.  However, only a few files are there (recent ones).

If files are not found on disk2, it falls-back to a PHP script which looks on 
disk1.  This is slow, but it allows us to view maps while new tiles are 
generated on the new disk.

If files are not found on disk2 or disk1, it falls-back to a PHP script which 
looks in the "blank sea tile" database.  I don't know the speed of this.

Obviously this scheme isn't fast enough, as the server code now has about 50% 
uptime

Options....

1: we could block access to the old disk. This means that PHP is never called 
for tile-viewing and everything goes through mod-rewrite

2: we could specifically request the whole world, so that the new disk fills. 
(I don't know if anyone is trying to do this already). 

3: we could remove old tiles from the exported list of tile metadata. This 
would cause various peoples' scripts to request any tiles where OSM has data 
but tiles at home doesn't. 

4: we could temporarily block access to blank-sea tiles (except for z-12, so 
that lowzoom script doesn't break), so that no tile-viewing requests use the 
database for a while.

5: We could copy images from the old disk to the new. This is probably the 
worst idea because (a) there's a lot of them, (b) many of them are outdated, 
and (c) they contain a lot of blank-sea images which we want to remove.

6: Someone could modify the mod-rewrite script [1] so that it looks on disk2, 
then if nothing found it looks on disk1, then if nothing found there either 
it calls the fallback PHP script to look for sea-tiles.

7: any other suggestions?


Currently I'm inclined to test option 1 first, to see what effect it has on 
performance.  

We should really do 2 or 3, so that we get stuff rendered asap, especially if 
it means that more requests go through mod-rewrite rather than PHP.



[1] http://svn.openstreetmap.org/sites/other/tilesAtHome/Tiles/.htaccess