[OSM-dev] render_expired for zoom 19 and metatiles

[Jon Burgess]

On Sun, 2011-01-16 at 20:08 +0100, Stephan Knauss wrote:
> During my Christmas vacation I set up a rendering up to zoom 19 on 
> thaimap.osm-tools.org
> 
> I recompiled mod_tile with MAX_ZOOM 19 (render_config.h)
> I set maxz in tirex backend config
> I even adjusted the maximum zoom in the munin scripts.
> 
> After that all looked quite nice.
> 
> Now I discovered my tile expiration not working as expected.
> I feed osm2pgsql expire list into render_expired.
> Inside render_expired the max zoom is also hard-coded to 18 in a lot of 
> places (at least the 18 appears more than once)
> 
> Question:
> Would changing all these magic numbers into 19 do the trick? I would 
> skip the twopow array. Or will I be breaking something not that obvious?

I'm afraid I can not answer that.

> Additional question:
> The more I look into the metatile format the more it is confusing me.
> Why is the -e parameter of osm2pgsql my maxzoom-3?
> This comment adds confusion:
>      // excess_zoomlevels is how many zoom levels at the large end
>      // we can ignore because their tiles will share one meta tile.
>      // with the default METATILE==8 this is 3.
> 
> Will metatiles really share tiles of *different* zoom levels?
> The different zooms have different styles in mapnik...
> 
> http://wiki.openstreetmap.org/wiki/Tirex/Internals#Metatile_File_Structure
> Does not list the zoom per entry. So all tiles inside a metatile would 
> be from the same zoom level.

Correct, a metatile is specific to one zoom

> Looking in my filesystem I also have directories for all zoom levels 
> down to 19, all containing meta files.
> 
> So it is more like a metatile contains 64 tiles of ONE zoom level.

Correct

> But if it's the later: How do these tiles get expired if all tools stop 
> 3 levels ahead?

What happens is that the tools track the exact (x,y) for all changed
tiles at one zoom level (say zoom 15). For all zooms less than 15 we can
calculate the expired tiles by halving the x & y for each step in zoom, 

e.g. a dirty tile at (x,y,z) = (512,600,15) at lower zooms...

(512,600,15) -> (256,300,14) -> (128,150,13) -> (64,75,12)...

The interesting thing is that because a metatile is an 8x8 grid of tiles
which we render all at once, we don't actually need the to track the
information of every single tile, just enough to hit one of the 8x8
sub-tiles of the meta tile. This means we can use our z15 information at
higher zooms too:

(512,600,15) -> (1024, 1200, 16) -> (2048, 2400, 17) -> (4096, 4800, 18)

You can step up by 3 zoom levels before this trick starts to break down
and the original tile that you were tracking would then begin to hit
16x16 higher zoom tiles = 2x2 grid of meta tiles. There is no reason why
you could not do this, but you would have to being interpolating the
(x,y) to expire multiple meta-tiles.


> I asked some questions regarding metatile removal before. In this thread 
> it was mentioned two times that the filesystem structure needs to be 
> adjusted for higher zoom levels. Why?

The existing meta-tile scheme builds the file path from a list of 5
bytes, "a/b/c/d/e.meta". These 5 bytes contain 40 bits of information
which is enough to hold data on 20 bits of X co-ordinates and 20 bits
for Y. This means the current scheme can only be used up to and
including zoom level 20. Beyond this it will break down. 

It is relatively easy to extend the current scheme to support more zoom
levels but we need to avoid breaking the existing setups which have
millions of tiles cached with the current directory layout. One simple
fix would be to extend to a 6 byte layout for zoom 21-24, 7 bytes for
zoom 25-28 etc. 

There are a couple of other minor tweaks which I'd like to make to the
directory layout and it would be better to make them all at once so that
we don't get a variety of schemes supported by different tools.

    Jon