[OSM-dev] Complete tileset - experiments with tile sizes and disk space

[Bill Magee]

Hi,

I'm new to OSM, please be gentle with me! I certainly do not wish to be 
an "architecture astronaut" as mentioned in the Wiki.

I've read through ...

<http://lists.openstreetmap.org/pipermail/dev/2008-April/009736.html>

... which is a thread regarding the filecount and filesize(s) involved 
to generate a complete planet tileset for OSM.

I understand why tiles are 256x256 pixels - for web delivery it makes 
sense, however for a locally deployed map the 256x256 tiles are quite 
inefficient for disk storage.

I don't do *nix but on WinXP the typical allocation size for large disks 
is 4kb (and I assume other file systems will do similar). Any tile under 
4kb (lots of them) will occupy this 4kb regardless.

I've rendered the planet through to zoom level 9 using the 
generate_tiles.py in both 256x256 and 2048x2048 tile sizes.

These figures are for the entire tile set from 0 through 9.

TileSize  Total Tiles Data Bytes Size on Disk
========= =========== ========== ============
256x256   552,329     220Mb      2,180Mb
2048x2048 8,841       140Mb      166Mb

There is still the possibility to remove blank tiles (either water or 
land) from either set, and certainly on the 2048px tiles I believe it 
would remove around 70%/45Mb on disk of the tiles. For display purposes 
I would have to maintain a list of whether a removed blank tile was 
water or land of course.

I have tried this for 2048px tiles at zoom level 11 and got the 
following,

Total Tiles Data Bytes Size on Disk
=========== ========== ============
13,433      722 Mb     749 Mb

That compares quite favourably with the 4million tile/21Gb theoretical 
or worst case total for zoom 11. Bear in mind that the 21Gb theoretical 
might actually be nearer 100Gb due to those 4kb allocations.

On the above numbers for zoom 11, I expect to fit a complete tileset to 
zoom 16 in under 1Tb. With low end 2Tb NAS boxes readily available for a 
few hundred pounds, I think this is a viable way forward (for me 
atleast!). In a few years time as disks continue to get bigger/cheaper, 
the entire planet to zoom 18 becomes deployable (I would expect a total 
of under 20Tb for this).

I imagine there will be an optimal tile pixel size in terms of disk 
space, and again in terms of what can be reasonably quickly rendered on 
a locally deployed tileset (at 2048px, the largest tile size is ~1Mb). 
Maybe that is something someone could experiment with.

Anyway, it is just an observation, and rambling thoughts/findings as to 
what sort of size a complete tileset could be scrunched down to.

 From my POV, I'm looking to offer OSM planet data alongside an existing 
application, and doing it with a complete tileset to z16 (even if it is 
~1Tb). This means deploying just data, no dependencies, no additional 
CPU load, no installs on customers production servers, no 
connectivity/bandwidth  requirements etc etc. Almost a fire and forget 
solution. I do understand that my requirements are not even close to 
those of OSM as a whole.

 From the OSM point of view, it may be possible to get your entire 
storage requirements for all zoom levels of the planet down to 20Tb - 
the worst case scenario where bandwidth is low, would be a slice and 
dice op for web delivery on the fly. Something like a mod_slice sitting 
alongside mod_tile.

Anyway, as I say, I'm new to OSM so this might be discussions you've 
already had - if so my apologies.

Kind Regards
Bill