[Tilesathome] another idea for more performance

[Damian Sulewski]

Hi,
ok i will try to explain it a little bit more, for all who want to know
how it works, and what i mean.

How it works in my understanding:
Server:
- a full tileset consists of 1365 tiles. For 2132 1358 at z12 (only some
data on it) 1154 blank tiles are uploaded (even though my
RenderFullTIleset = 0) 
- Meta info is stored in tiles_meta table, for each uploaded tile. If
the  client uploads a full tileset only data for  zoom 12 is replaced
(thats why a full tileset _must_ have 1365 tiles, all must "belong" to
the same uploader)
- For every _uploaded_ blank tile, an entry in tiles_blank is made (with
a replace mysql order)
- For every _uploaded_ tile an existing file on disk would be deleted.
- For every _uploaded_ tile the meta info will be deleted (code not on
the server by now, but in svn track: 3017)
- If a tile is requested the server looks at the disk, if not found
there it looks a the specified position in tiles_blank, if not found
there it looks up recursively at ahigher zoom level until something is
found.

Client:
- Tries to upload only full tilesets.
- Because of 2mb limit on server, it can upload tiles as a "non full"
tileset, splitting a tileset in several chunks.
- If the whole area z12 is empty, it uploads only one blank tile,
calling it a full tileset, and zipping it. 

Problems:
- for the example "2132 1358" , 1154 REPLACE orders are taken in
tiles_blank, although much less would be needed because of recursive
lookup
- "blank land" is 67 bytes, compressed blank land is 104 bytes
- for every tileset,also for the ones, with only one tile in it, the
server: generates a dir, unzips the tile, reads the tile from disk,
processes it, deletes the dir

Optimization:
- Frederiks "sequential processing" will help. The problems above will
still remain, but they will be trated faster. 
- my aproach: do not upload a whole tileset, just upload the blank at
the highest possible position. The server must make sure, all meta-info
and files are deleted under this blank tile.
- - still processing each tile but a loop on the server replacing each
tile, is faster then unzipping the tile, reading it from disk, and
processing it.
-another idea: if a tileset contains only one tile, upload the tile. the
server should handle this, and just insert it into the db, no zip, no hd
access. Still deleting all tiles under it.

This is a rather long one, i hope someone came to read the mail until
here ;-)

Now the replies:

> Yes. That's what I meant when I said "to my knowledge it is not 
> implemented yet", and at the same time I suggested doing this deletion 
> not when(ever) a blank tile is uploaded, but instead asynchronously. 
i sugested this on IRC, but there were rumors, that the maplint layer
needs immediate deletion.

> I 
> suspect that the whole "doing things when a tile is uploaded" chain is 
> already stretched a bit too far; remember that anything we do "when a 
> tile is uploaded" is done by PHP code while the Apache process holds the 
> HTTP connection to the uploading client, and there may be a lot of 
> parallel uploads. Any functionality we manage to move out of this 
> processing and into some kind of regular maintenance process (of which 
> there's only one instance and not 20 parallel ones fighting for 
> ressources) is probably good.

I fully agree with you. But even if there would be only one process
running, we should optimize it. Imho, optimizing the upload process now,
gives us a better start for sequential prozessing of the tilesets.

Greetings,

Damian

-- 
Verification of concurrent C++ programs:
http://StEAM.cs.uni-dortmund.de