[Tilesathome] Proposal: Keeping tileset as one file

[Sebastian Spaeth]

Jiri Klement wrote:
>>  Instead of proclaiming a bottleneck, why not show us?
> 
> You can test on t at h server how much faster is copying one big file
> than hounders of small files.

No doubt that this file operation will be quicker, but is that all we'd 
need to test for? We need to 1)Upload (no change) 2)Check if the tileset 
is complete (ie, peek at every tile in the uploaded blob) 3a) move the 
tile to its end destination (if it's a complete tileset) 3b) or extract 
the tiles and insert them into the destination blob if only a single 
tile, or a few tiles, or an incomplete tileset has been uploaded (means 
copying over the whole binary blob, and inserting the new tiles into it)

I'd expect that especially 3b is much worse in the tileset-per-file system.

And then you have to consider tile serving. Right now, it's one file per 
tile, no PHP etc involved. Apache and the OS can do all their file 
system tricks, e.g. caching often viewed z12 tiles in memory etc.
The one-tileset-per-file approach would require you to open the 
tilesetfile, extract the data blob, wrap it into a PNG structure, and 
serve it. If I browse the z12 layer, I don't see any caching advantages, 
as rather than opening one file per tile, it has to open one tileset 
file per tile and deliver parts of it. I'd expect poorer caching results 
as the bigger tileset files fall out of memory quicker, not better ones. 
Admittedly at z13, you'd only have to open one file for every 2 z13 
tiles and it get's better at higher zoom levels. Still you would have to 
cache whole tilesets rather than just the often-used tiles in memory, so 
there are arguments for both sides.

> My tests shows that copying 100 tilesets (700kB, 138 actuall tiles)
> over nfs on LAN needs 8 seconds for 1 file per tileset and 65 seconds
> for one file per tile.

As we are getting away from NFS connected disks in favor of local disks 
pretty soon now (hopefully), I wouldn't recommend optimizing the whole 
thing for NFS access.

I agree that unzipping a tileset with 1600 disk seeks is more expensive 
than just pushing one big file. I still haven't been convinced though 
that *serving* tiles, would be quicker this way though. Actually, the 
whole tile serving seems to perform pretty well, I haven't heard of any 
bottlenecks. And it's pretty simple, which is a plus.


I don't think it's very obvious or easy to claim the superiority of one 
system over the other. The existing system has the advantage that, errr, 
it exists. :-) So, I back up deelkar here, "show us" and we'll be happy 
to use the better performing system.

spaetz