[Tilesathome] Proposal: Keeping tileset as one file
Sebastian at SSpaeth.de
Mon Jun 9 08:32:59 BST 2008
Jiri Klement wrote:
>> Instead of proclaiming a bottleneck, why not show us?
> You can test on t at h server how much faster is copying one big file
> than hounders of small files.
No doubt that this file operation will be quicker, but is that all we'd
need to test for? We need to 1)Upload (no change) 2)Check if the tileset
is complete (ie, peek at every tile in the uploaded blob) 3a) move the
tile to its end destination (if it's a complete tileset) 3b) or extract
the tiles and insert them into the destination blob if only a single
tile, or a few tiles, or an incomplete tileset has been uploaded (means
copying over the whole binary blob, and inserting the new tiles into it)
I'd expect that especially 3b is much worse in the tileset-per-file system.
And then you have to consider tile serving. Right now, it's one file per
tile, no PHP etc involved. Apache and the OS can do all their file
system tricks, e.g. caching often viewed z12 tiles in memory etc.
The one-tileset-per-file approach would require you to open the
tilesetfile, extract the data blob, wrap it into a PNG structure, and
serve it. If I browse the z12 layer, I don't see any caching advantages,
as rather than opening one file per tile, it has to open one tileset
file per tile and deliver parts of it. I'd expect poorer caching results
as the bigger tileset files fall out of memory quicker, not better ones.
Admittedly at z13, you'd only have to open one file for every 2 z13
tiles and it get's better at higher zoom levels. Still you would have to
cache whole tilesets rather than just the often-used tiles in memory, so
there are arguments for both sides.
> My tests shows that copying 100 tilesets (700kB, 138 actuall tiles)
> over nfs on LAN needs 8 seconds for 1 file per tileset and 65 seconds
> for one file per tile.
As we are getting away from NFS connected disks in favor of local disks
pretty soon now (hopefully), I wouldn't recommend optimizing the whole
thing for NFS access.
I agree that unzipping a tileset with 1600 disk seeks is more expensive
than just pushing one big file. I still haven't been convinced though
that *serving* tiles, would be quicker this way though. Actually, the
whole tile serving seems to perform pretty well, I haven't heard of any
bottlenecks. And it's pretty simple, which is a plus.
I don't think it's very obvious or easy to claim the superiority of one
system over the other. The existing system has the advantage that, errr,
it exists. :-) So, I back up deelkar here, "show us" and we'll be happy
to use the better performing system.
More information about the Tilesathome