[Tilesathome] Observations after having played around with t at h server

[spaetz]

On Fri, Jul 27, 2007 at 08:45:21AM +0200, John Bäckstrand wrote:
> Ok, I wanted to play around with the server and try to see where the 
> bottlenecks laid with regards to queue processsing. What I did was 
 
> maplint tiles are never "fast", because the check in lib/checkupload.inc 
> checks against the number of tiles for z17 which maplint doesnt contain 
> atm, it only goes to z16. Maplint tilesets will always be flagged as 
> incomplete. These example sets have very fet metainserts anyway (0-32 
> for my examplezips) so making the take the FAST path would not really 
> make a difference anyway I think, compared to the 341 blank deletes.

I've seen the number 341 a lot too, now I now what it means, thanks :-).
Seriously, that is something we might want to consider. Having a configurable number of files for a full tileset per layer might help a lot. I still haven't looked at the individual vs tileset handling, so don't know how easy that would be.
 
> Secondly, there are alot of processing done for tiles_blank, especially 
> for maplint since these will ideally be blank! I have an optimizing 
> hashing approach almost done, but I dont think it will make a huge 
> difference, see below.

Yes blank tile handling could definitly do with a bit of improvement. One thing I could imagine is checking whether the tile is blank and not update/replace the database in that case at all. Currently the disk with our data bases is still spinning a lot (looking at IOstat), so reducing writes will still help us.

> Thirdly, on my laptop its not like the _entire_ DB access is a low 
> hanging fruit at all. My disk is arguably very slow imo. Removing all DB 
> access alltogether takes the time down to 10 seconds from 19s. Thats 
> still "only" one tileset per second. Initially I stored all tiles on the 
> disk, and the times for 10 zips was ~65 seconds, moving to a RAM-disk 
> brought this down to 30 seconds.

Nice, unzipping files to RAM disk would be good. However on this machine with 1GB RAM and two data abses running and apache running, I think, we can't afford big RAM disks. There are plans to upgrade dev's memory though. Tiles are just moved on our disk within one partition, so the moves should be really quick.

> I think we might have to rethink how this is all done. The most optimal 
> way to do things that I can see is to let the client actually know whats 
> on the server (per each tile, whether it is blank land, sea and if 
> regular the last hash of the png?

More intelligent clients giving less work (ie omitting non-changed tiles) would definitely be the biggest boost of all possible things, but also the most intrusive ones. One  relatively easy fix could be, if the t at h server gives the renderer the date of the oldest tile it has in that tileset. Than the renderer could check the last modification date of the OSM file in question. If the oldest tile is way newer than the last modification date, the whole tileset shouldn't be rendered and could be returned empty.

I guess deelkar and ojw will have to say something about feasability.
spaetz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/tilesathome/attachments/20070727/e3120b7f/attachment.pgp>