[Tilesathome] Observations after having played around with t at h server

[John Bäckstrand]

Dirk-Lüder Kreie wrote:
> I'll have a look at that checkupload.inc
> If it really does check wether all files are present, then *please* do
> tell that to client devs, because right now (since February I believe)
> there is a nice little optimization that cuts short tile rendering if
> there is nothing to render. so the subtiles of a completely empty tile
> will not be created. If it's necessary for the check that all files be
> present just one config switch needs to be thrown in the client by
> anyone with svn access. This will then mean you get every tile rendered
> and every time an empty(land/sea) tile instead of getting nothing there now.

The checkupload result only has an impact on whether we do metadata 
inserts per tile or per z12 tileset. Blank tiles are still always 
handled per-tile, so not uploading those files will very likely result 
in quicker processing, since there is no shortcut on the server for 
blank tiles: every blank tile in the zip is handled every time.

This means that tilesets with a large amount of blanks in them will 
actually benefit from the client-side optimization. For a crowded, big 
tileset the metadata inserts might be slower, I did not test this.

> As long as you handle one zip at a time the Ramdisk will need to be in
> the 2-digit MB range, see my stats about uploaded zips, and take times 2
> (pngs are just stored in zips) or maybe 2.5 because of block size overhead?
> Currently the limit would be 10 MB per zip anyway (upload limit).

I have to point out that I "cheated" a bit here. I store the incoming 
zip, the temporary files and the outgoing tiles in my ramdisk. (For some 
reason the disk is very busy still, might be the DB accesses). So the 
gains in production will likely be much less. I don't think we want to 
store the queue in RAM, and definitely not the resulting tiles ;)

The only thing a temp-folder-in-ram will gain us is a litte bit of seek 
times I guess. I am not sure how the access-patterns for the actual 
tiles look like, I think we mostly traverse the filenames.

> Here's an updated statistic about zip file sizes on my archive disk:
> 
> between 0 and 1 MB: 21165
> between 1 and 2 MB: 1573
> between 2 and 3 MB: 386
> between 3 and 4 MB: 233
> between 4 and 5 MB: 109
> between 5 and 6 MB: 89
> between 6 and 7 MB: 44
> between 7 and 8 MB: 28
> between 8 and 9 MB: 39
> between 9 and 10 MB: 10
> between 10 and 11 MB: 9
> between 11 and 12 MB: 9
> between 12 and 13 MB: 9
> between 13 and 14 MB: 1
> between 14 and 15 MB: 8
> between 15 and 16 MB: 2
> between 16 and 17 MB: 3
> between 17 and 18 MB: 2
> between 18 and 19 MB: 2
> between 19 and 20 MB: 0
> between 20 and 21 MB: 0

My zips are all very small (all but one is <1MB, which is 2.3MB) and 
very "empty". I would be glad to receive more sample zips of varying 
sizes? I do think that anything above 5MB is probably not worth 
optimizing much, since they account for a very small portion of all 
tiles? Could still be interesting to have.

This statistics is very interesting though, it makes it possible to 
actually estimate the final result of tweaks: for example, tweaks to 
handle 0-1MB tiles should have a much greater impact that those for 10MB 
tiles.

I would also like to have multiple variants of the same z12 tileset, 
where the subset of blank tiles have in some way changed, ie. a 
previously blank tile became unblank, or the other way around. I could 
produce these myself, but if anyone have anything like this laying around :)

---
John Bäckstrand