[OSM-dev] Complete tileset - experiments with tile sizes and disk space
bill at billmagee.co.uk
Mon Dec 22 13:43:27 GMT 2008
I'm new to OSM, please be gentle with me! I certainly do not wish to be
an "architecture astronaut" as mentioned in the Wiki.
I've read through ...
... which is a thread regarding the filecount and filesize(s) involved
to generate a complete planet tileset for OSM.
I understand why tiles are 256x256 pixels - for web delivery it makes
sense, however for a locally deployed map the 256x256 tiles are quite
inefficient for disk storage.
I don't do *nix but on WinXP the typical allocation size for large disks
is 4kb (and I assume other file systems will do similar). Any tile under
4kb (lots of them) will occupy this 4kb regardless.
I've rendered the planet through to zoom level 9 using the
generate_tiles.py in both 256x256 and 2048x2048 tile sizes.
These figures are for the entire tile set from 0 through 9.
TileSize Total Tiles Data Bytes Size on Disk
========= =========== ========== ============
256x256 552,329 220Mb 2,180Mb
2048x2048 8,841 140Mb 166Mb
There is still the possibility to remove blank tiles (either water or
land) from either set, and certainly on the 2048px tiles I believe it
would remove around 70%/45Mb on disk of the tiles. For display purposes
I would have to maintain a list of whether a removed blank tile was
water or land of course.
I have tried this for 2048px tiles at zoom level 11 and got the
Total Tiles Data Bytes Size on Disk
=========== ========== ============
13,433 722 Mb 749 Mb
That compares quite favourably with the 4million tile/21Gb theoretical
or worst case total for zoom 11. Bear in mind that the 21Gb theoretical
might actually be nearer 100Gb due to those 4kb allocations.
On the above numbers for zoom 11, I expect to fit a complete tileset to
zoom 16 in under 1Tb. With low end 2Tb NAS boxes readily available for a
few hundred pounds, I think this is a viable way forward (for me
atleast!). In a few years time as disks continue to get bigger/cheaper,
the entire planet to zoom 18 becomes deployable (I would expect a total
of under 20Tb for this).
I imagine there will be an optimal tile pixel size in terms of disk
space, and again in terms of what can be reasonably quickly rendered on
a locally deployed tileset (at 2048px, the largest tile size is ~1Mb).
Maybe that is something someone could experiment with.
Anyway, it is just an observation, and rambling thoughts/findings as to
what sort of size a complete tileset could be scrunched down to.
From my POV, I'm looking to offer OSM planet data alongside an existing
application, and doing it with a complete tileset to z16 (even if it is
~1Tb). This means deploying just data, no dependencies, no additional
CPU load, no installs on customers production servers, no
connectivity/bandwidth requirements etc etc. Almost a fire and forget
solution. I do understand that my requirements are not even close to
those of OSM as a whole.
From the OSM point of view, it may be possible to get your entire
storage requirements for all zoom levels of the planet down to 20Tb -
the worst case scenario where bandwidth is low, would be a slice and
dice op for web delivery on the fly. Something like a mod_slice sitting
Anyway, as I say, I'm new to OSM so this might be discussions you've
already had - if so my apologies.
More information about the dev