[Openstreetmap-dev] Landsat Disk Cache Setup

Mikel Maron mikel_maron at yahoo.com
Tue Jan 17 11:47:16 GMT 2006


I'm setting up the Landsat cache, and want to sanity check some things with you all, especially those more expert in sysadmin than myself. Questions on setting up squid are below, first some calculations. 
 
ESTIMATED REQUIREMENTS: 
 
landsat.openstreetmap.org (dev) has a 110 Gig drive. Currently only 5% in use. The maximum suggested disk allocation for squid is about 60%. 50Gb is below this, will be large enough to be effective, and leaves plenty of room for other purposes. 
 
A complete cache of landsat tiles, including all instrument bands, is several terrabytes. There is a 300 Gb cache in the works by OnEarth, using only a preprocessed selection of visual bands, but it's unclear when that will be widely available.  
 
Currently composted tiles are cached down to a maximum of 20 levels of zoom. With landsat's imagery resolution, its actually only necessary to request and cache tiles to 12 levels of zoom. For higher zoom levels, clients can simply request tiles cached at level 12 and scale them appropriately (I've added a ticket for this).  
 
My calculation for storing these images as JPEGS.. 
 
16 [initial number of tiles] * 2 ** (12 * 2) [number of tiles at 12 levels of zoom] * 1.33333 [Sum of the geometric series of higher zoom levels] * .3 [30% of earth covered by land] * 6k [average tile size] = ~650 Gb. 
 
50 Gb will store 1/13 of those tiles, say 1/15 to be safe, since there's small overhead in squid and some variation in each tile. OpenStreetMap activity is concentrated in a much smaller portion of the earth's surface. I haven't calculated that from the current database of vectors and gpx tracks, but it is certainly much smaller than 1/15 of the earth's land surface. We'll be doing pretty well with this cache. 
 
SQUID SETUP: 
 
Squid caches objects in a two tier directory. It's recommended that the level 2 tier always contain 256 directories. The number of level 1 directories is calculated as 
 
(((50MB [cache size in KB] / 6 [average object size in KB]) /256) /256/) *2 = ~ 267 
 
 So squid would create 256 * 267 directories. With 50 G cache, and average 6k file size, the maximum number of inodes is something like 8 million.   The disk reports over 14 million inodes available. So it looks like that's within limits.

Does this sound sane?

Thanks
Mikel


 






More information about the dev mailing list