[Tilesathome] Proposal: Keeping tileset as one file

[Florian Lohoff]

On Mon, Jun 09, 2008 at 05:15:33PM +0200, Florian Lohoff wrote:
> http://silicon-verl.de/home/flo/tmp/tahbench-20080609.tgz
> 
> to play with. Its by no means optimized or fixes all problems but it
> should show that the millions of little files approach is a performance
> problem. BTW: With 4K blocks in the filesystem one wastes an average
> of 2K per file which with my working set (~140K files) this sums up to
> 280Mbyte wasted space on disk but even worse - worse cache footprint.
> With 16kbyte average file size 12% of disk space is wasted because of
> the last block beeing statistically half empty.

Just as a coincidence - Slashdot is running a story about the Facebook
photo storage. Basically they want to do the same as this. Storing all
the photos in large chunks with external index. They even encode the
start and length of the file to serve within the url (and some security
measures).

http://beta.flowgram.com/f/p.html#2qi3k8eicrfgkv

So - if they start serving 450TByte with 6.5 Billion photos like this 
it cant be all so wrong.

Flo
-- 
Florian Lohoff                  flo at rfc822.org             +49-171-2280134
	Those who would give up a little freedom to get a little 
          security shall soon have neither - Benjamin Franklin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20080625/a2c37dd1/attachment.pgp>