[Tilesathome] Observations after having played around with t at h server

[Dirk-Lüder Kreie]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

spaetz schrieb:
> On Fri, Jul 27, 2007 at 08:45:21AM +0200, John Bäckstrand wrote:
>> Ok, I wanted to play around with the server and try to see where the 
>> bottlenecks laid with regards to queue processsing. What I did was 
>  
>> maplint tiles are never "fast", because the check in lib/checkupload.inc 
>> checks against the number of tiles for z17 which maplint doesnt contain 
>> atm, it only goes to z16. Maplint tilesets will always be flagged as 
>> incomplete. These example sets have very fet metainserts anyway (0-32 
>> for my examplezips) so making the take the FAST path would not really 
>> make a difference anyway I think, compared to the 341 blank deletes.
> 
> I've seen the number 341 a lot too, now I now what it means, thanks :-).
> Seriously, that is something we might want to consider. Having a configurable number of files for a full tileset per layer might help a lot. I still haven't looked at the individual vs tileset handling, so don't know how easy that would be.

I'll have a look at that checkupload.inc
If it really does check wether all files are present, then *please* do
tell that to client devs, because right now (since February I believe)
there is a nice little optimization that cuts short tile rendering if
there is nothing to render. so the subtiles of a completely empty tile
will not be created. If it's necessary for the check that all files be
present just one config switch needs to be thrown in the client by
anyone with svn access. This will then mean you get every tile rendered
and every time an empty(land/sea) tile instead of getting nothing there now.

[...]
>> Thirdly, on my laptop its not like the _entire_ DB access is a low 
>> hanging fruit at all. My disk is arguably very slow imo. Removing all DB 
>> access alltogether takes the time down to 10 seconds from 19s. Thats 
>> still "only" one tileset per second. Initially I stored all tiles on the 
>> disk, and the times for 10 zips was ~65 seconds, moving to a RAM-disk 
>> brought this down to 30 seconds.
> 
> Nice, unzipping files to RAM disk would be good. However on this machine with 1GB RAM and two data abses running and apache running, I think, we can't afford big RAM disks. There are plans to upgrade dev's memory though. Tiles are just moved on our disk within one partition, so the moves should be really quick.

As long as you handle one zip at a time the Ramdisk will need to be in
the 2-digit MB range, see my stats about uploaded zips, and take times 2
(pngs are just stored in zips) or maybe 2.5 because of block size overhead?
Currently the limit would be 10 MB per zip anyway (upload limit).

Here's an updated statistic about zip file sizes on my archive disk:

between 0 and 1 MB: 21165
between 1 and 2 MB: 1573
between 2 and 3 MB: 386
between 3 and 4 MB: 233
between 4 and 5 MB: 109
between 5 and 6 MB: 89
between 6 and 7 MB: 44
between 7 and 8 MB: 28
between 8 and 9 MB: 39
between 9 and 10 MB: 10
between 10 and 11 MB: 9
between 11 and 12 MB: 9
between 12 and 13 MB: 9
between 13 and 14 MB: 1
between 14 and 15 MB: 8
between 15 and 16 MB: 2
between 16 and 17 MB: 3
between 17 and 18 MB: 2
between 18 and 19 MB: 2
between 19 and 20 MB: 0
between 20 and 21 MB: 0

Total: 23727 Tileset zips
plus 2791 split zips (chunks)

- --

Dirk-Lüder "Deelkar" Kreie
Bremen - 53.0952°N 8.8652°E

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGqbDDFUbODdpRVDwRAjtMAJ9+VUAcamoL5rZ2iV6NHfITiZVgeACgpYzB
zUY6d9V+ZACR7xa/F2xCZRQ=
=+yc3
-----END PGP SIGNATURE-----