[Tilesathome] Tileset-at-once - we can now upload 8MB ZIP files

[Dirk-Lüder Kreie]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dirk-Lüder Kreie schrieb:
> OJW schrieb:
>> Sebastian has increased the size of ZIP files which can be uploaded (to 8MB I 
>> think*), so we can start uploading "one tileset per upload" much more often.
> 
>> * http://dev.openstreetmap.org/~ojw/Upload/info.php
> 
>> (1) I noticed that the tiles at home client in SVN now has the limit changed to 
>> 10MB -- do we need to publicise this more?  
> 
>> (2) Do we need a new version name, so that we can spot people who are still 
>> uploading 2-8MB tilesets in 2MB blocks unnecessarily?
> 
>> (3) There is also a suggestion that the limit should be more like 20MB (so 
>> that even the largest tilesets can be uploaded in one go) - any comments on 
>> that idea?
> 
> I have run some statistics on my archive of uploaded and oversized tilesets:
> 
> between 0 and 1 MB: 19139
> between 1 and 2 MB: 1316
> between 2 and 3 MB: 297
> between 3 and 4 MB: 188
> between 4 and 5 MB: 85
> between 5 and 6 MB: 72
> between 6 and 7 MB: 36
> between 7 and 8 MB: 18
> between 8 and 9 MB: 30
> between 9 and 10 MB: 8
> between 10 and 11 MB: 8
> between 11 and 12 MB: 7
> between 12 and 13 MB: 8
> between 13 and 14 MB: 0
> between 14 and 15 MB: 4
> between 15 and 16 MB: 1
> between 16 and 17 MB: 0
> between 17 and 18 MB: 2
> between 18 and 19 MB: 0
> between 19 and 20 MB: 0
> 
> 21224 tilesets total
> 
> the tilesets larger than 2 MB were converted into 2574 additional split
> up zips.
> 
> I think the limit should be ~20MB (memory permitting) so we can force
> only tileset uploads, which will enormously reduce metadata overhead.
> 
> for this lowzoom.pl needs to be adapted to process z0 and z6 from z6 and
> z12 preferably.

Note: these encompass maplint tilesets as well, which are mainly much
smaller, but same in count as the normal tileset zips, so this statistic
is a bit skewed towards the low filesizes.

If you turn this into a sort of optimization problem and postulate that
the average tileset contains a 1000 Tiles and that a split tileset thus
costs the server 1000x as much metadata updates you need to find the
point where its more cost effective to split up the zip instead of
accepting very large files, which would be, looking at the metadata the
point where less than 1 in 1000 zips are split up.
That would be just about 12 MB if I count right.
(0+0+2+0+1+4+0+8) / 21224 < 1/1000
(0+0+2+0+1+4+0+8+7) / 21224 > 1/1000

OR if you want to get rid of any freeform uploads (that means converting
lowzoom to using some sort of "tileset" mechanism itself) you need to
find the absolute maximum and set the upload limit there, which in turn
should greatly decrease the need for kept metadata.

- --

Dirk-Lüder "Deelkar" Kreie
Bremen - 53.0952°N 8.8652°E

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGoSm1FUbODdpRVDwRAqceAJ9ZB9pgV4zF0uJdIbpr/wKCRIy/wgCfTfzL
SPvVWzIQzdhiw+BBSrt47ck=
=nv3T
-----END PGP SIGNATURE-----