[OSM-dev] Slippy Map, automatic Tile Rendering

[Frederik Ramm]

Hi all,

    I've been looking at the "tiles at home" slippy map. For those of  
you who are not familiar with how tile generation for the slippy map  
works (as I was a week ago, before I started a barrage of "why" and  
"how" emails aimed at Oliver White), here's a short overview. And  
after that a few questions.

The zoom levels in use are 7 (128x128 tiles for the whole world) to  
17 (most detailed). Increasing the zoom level by 1 gives you 2x2  
tiles from every 1 tile you had before.

For zoom levels 12 to 17, individual osmarender styles have been  
defined, so the low-detail levels do not render road names etc. - The  
tiles at home architecture can be used re-generate tiles on any zoom  
level, however it will always go down to a configured most-detail  
zoom level (normally 17). I.e. if you request the rendering of a  
certain level-12 tile on tiles at home, you will get one level-12 tile,  
four level-13 tiles, 16 level-14 tiles, etc.

There are currently about 1.6 million level-12 tiles in the database,  
and 6.6 million tiles total (this tells us that many level-12 tiles  
have been generated without going down to level-17 detail, otherwise  
there should be a total of roughly 2 billion tiles for 1.6 million  
level-12 tiles). Many of the level-12 tiles are, however, blank; they  
seem to stem from a now discontinued approach where level-12 tiles  
were chosen randomly and rendered "just in case" there was data.

Zoom levels 7 to 11 are generated differently; for those, the  
level-12 tiles are pasted together and scaled. This is done by an  
utlity script in the tiles at home dir called "lowzoom". Called with the  
co-ordinates of a specific level-7 tile, it will collect the 1024  
level-12 tiles making up that level-7 tile and create 256 level-11  
tiles, 64 level-10 tiles, 16 level-9 tiles, 4 level-8 tiles and one  
level-7 tile by scaling down.

(That's the reason the low-detail zoom levels look so faint; on the  
other hand, if one were to define modified stylesheets for these  
levels, it would be next to impossible to download the amount of osm  
data needed to render one because of their large bounding box. This  
will probably only be possible if the OSM API did support some sort  
of filtering.)

The "lowzoom" process, i.e. generating the level-7 to level-11 tiles,  
is not fully automatic. A full list of tiles on the server is  
available from http://dev.openstreetmap.org/~ojw/Stats/. This list  
can be used to exctract a list of all non-blank level-12 tiles, and  
from that one can find out which level-7 tiles actually warrant  
generating because at least one level-12 tile lies in their area. It  
turns out there are only about 280 level-7 tiles  (of a possible  
16,384 convering the earth) for which non-blank level-12 tiles exist;  
re-generating them (and with them, the intermediate levels 8,9,10,11)  
takes only about half a day, so this is not much of a concern and  
I'll probably set something up to run that automatically.

It is more time-consuming and complicated however to create level-12  
tiles. There are about 50 contributors who currently render tiles for  
tiles at home. Requests for tile generation are issued by the system if  
either someone clicks the "request update" button in the tile  
browser, or if activity on a certain tile has been detected by  
monitoring the RSS feed.  This, however, misses a lot of edits,  
because only the last 40 edits are contained in the feed and as soon  
as more edits occur between two feed requests, they go unnoticed.

As a result, many of the level-12 tiles on the server are outdated  
and will be outdated in the future. I am looking for ways to (a)  
determine which tiles are outdated to have them re-rendered, and (b)  
do that regularly in the future.

(a) Which tiles are outdated? One could of course just re-generate  
all existing tiles but that would be very time-consuming (I guess  
about 20 days of round-the-clock computing for a modern PC; less of  
course if many take part in the distributed computing). I will try to  
evaluate the last-change-timestamp in the planet file and use that to  
determine which tiles are out of date.

(b) If I want to regularly find outdated tiles, the perfect way would  
of course be to process a full change log instead of the limited RSS  
feed. Ideally, I would want to ask the server "please send all nodes  
changed or added since <timestamp>", then use this regularly to re- 
generate all affected tiles. Does someone know if a feature like this  
is planned or even likely? - The other approach is,  again, using the  
weekly planet files to find out which nodes have changed. But this  
would lead to weekly updates only.

I am not saying that we should abandon the current tile regeneration  
based on the RSS feed, but it seems to me that it needs to be  
supplemented by something else if it alone cannot guarantee that our  
tiles are halfway current.

Any comments?

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00.09' E008°23.33'