[OSM-dev] [OSM-talk] Mapnik tileset coherency issues

Tom Hughes tom at compton.nu
Fri Oct 19 08:44:50 BST 2007

In message <47182550.4030208 at deelkar.net>
        Dirk-Lüder Kreie <osm-list at deelkar.net> wrote:

> What we learned from T at H that storing tiles in blobs for non-static
> tiles is a bad idea, because the tiles tend to grow in complexity and
> thus filesize, so the table apparently becomes fragmented.

MySQL is definitely a bottleneck for the mapnik layer as well, hence
the plan to move to the filesystem with mod_tile.

> Storing tiles in a filesystem is not too great either, if you just stick
> to paths like ./z/x/y.png.
> Instead you should split the paths up further so there aren't that many
> files in the directories, and hide that fact with a mod_rewrite rule, so
> you get nice "like google" URLs back for the tiles.
> For example the tile 130543,80370 zoom17 is stored as
> ./17/130/543/080/370.png
> ensuring that no directory has more than 1000 entries.

So long as you are using indexed directories it shouldn't be too much
of a problem to have so many files in a directory, and recently created
ext3 filesystem should be using indexed directories.

A more interesting question is choosing a filesystem that doesn't
waste too much space for lots of small files - it looks like the
tile filesystem on dev is using reiserfs which was my first thought
as well. Jon reckoned that although it did waste less space it was
slower as a result because of the time spent trying to find a hole
of the right size?

> As for marking tiles "dirty" by proximity: if you start doing it for
> Zoom direction, only do it for *lower* zooms, or you will get into
> trouble pretty fast, as each higher zoom layer adds 3n+1 tiles, so more
> than triples the work to be done for each zoom added.

I'm not sure that implicit rendering of lower zooms helps though, but
implicit rendering of higher zooms probably does because you're quite
like to zoom in on some detail of the current view but much less likely
to zoom out.

With mapnik at least the time taken to render tiles get significantly
less the more you zoom in, so low zoom tiles are more expensive to render.

> As already said, the tiles should only be marked "dirty" if not rendered
> since last mapnik db update (i.e. last planet/planetdiff import).

This is already supported in the mod_tile work.

> At some point in the future you might want to give up on storing each
> tiles metadata individually, but grouping tiles from several layers
> together to "tilesets" which get marked "dirty" in one go, (i.e. 18-16,
> 15-13, 12-10, each 21 tiles) which will automatically give some manner
> of z-coherency and xy-coherency at the higher zooms.

There isn't any metadata in the mod_tile scheme - the only information
it uses is the modification time of the file to know when it was last
rendered. The dirty tile queue is (currently at least) held in memory
by the render daemon.


Tom Hughes (tom at compton.nu)

More information about the dev mailing list