[OSM-dev] Tiles at home - current disk use
streetmap at blibbleblobble.co.uk
Sat May 19 14:07:36 BST 2007
On Friday 18 May 2007 13:09, SteveC wrote:
> but the new drive isn't actualy being used
OK, let's explain what's happening with disks:
Background info 1: "disk1" filled up, and write performance was adversely
affected by using a disk at near its capacity limit.
Background info 2: Early versions of land/sea rendering created a lot of blank
blue tiles which were stored as images. We'd prefer to store blank tiles as
a database entry (or binary file, etc), but detecting and removing the old
ones takes a lot of CPU and disk access on the server
Background info 3: The new land/sea rendering processes means that it's
desirable to have re-renders of everything, so that coastlines will show up
on the map.
New uploads are all being saved to "disk2". Disk2 is amost empty, and has
more than enough space to store all the tiles that rendering the current OSM
dataset will generate. Eventually this scheme will need to be replaced with
one which distributes data across both disks, but we have some time to think
about that rather than doing it now without prior testing.
I've asked a couple of people to try requesting re-renders of everything.
Basically, look for tiles generated before 2007-05-19 and request them.
mod_rewrite looks on disk2. Tiles on disk2 will therefore be faster to
access. However, only a few files are there (recent ones).
If files are not found on disk2, it falls-back to a PHP script which looks on
disk1. This is slow, but it allows us to view maps while new tiles are
generated on the new disk.
If files are not found on disk2 or disk1, it falls-back to a PHP script which
looks in the "blank sea tile" database. I don't know the speed of this.
Obviously this scheme isn't fast enough, as the server code now has about 50%
1: we could block access to the old disk. This means that PHP is never called
for tile-viewing and everything goes through mod-rewrite
2: we could specifically request the whole world, so that the new disk fills.
(I don't know if anyone is trying to do this already).
3: we could remove old tiles from the exported list of tile metadata. This
would cause various peoples' scripts to request any tiles where OSM has data
but tiles at home doesn't.
4: we could temporarily block access to blank-sea tiles (except for z-12, so
that lowzoom script doesn't break), so that no tile-viewing requests use the
database for a while.
5: We could copy images from the old disk to the new. This is probably the
worst idea because (a) there's a lot of them, (b) many of them are outdated,
and (c) they contain a lot of blank-sea images which we want to remove.
6: Someone could modify the mod-rewrite script  so that it looks on disk2,
then if nothing found it looks on disk1, then if nothing found there either
it calls the fallback PHP script to look for sea-tiles.
7: any other suggestions?
Currently I'm inclined to test option 1 first, to see what effect it has on
We should really do 2 or 3, so that we get stuff rendered asap, especially if
it means that more requests go through mod-rewrite rather than PHP.
More information about the dev