[Tile-serving] OSM Tile Expiration

[Sarah Hoffmann]

On Sat, Jul 25, 2020 at 09:13:51AM +0200, Christian Quest wrote:
> Le 24/07/2020 à 22:50, Sarah Hoffmann a écrit :
> > The OSMF tile script only uses change files and the flatnode store to do
> > the expiry. That means that it does not catch deletes or when something is
> > moved out of a tile. (It only sees the situation after the update was
> > applied.) It also does not care about relations at all. So it might miss
> > a tile that is enitrely within a multipolygon and does not catch boundary
> > changes that just consist of members being added and deleted.
> > 
> > osm2pgsql's expiry catches all these cases at the price of additional
> > database accesses. Also, the whole implementation is not exactly efficient.
> > We have a long-standing feature request to overhaul that piece of code.
> 
> 
> Thank you Sarah, it confirms my understanding of the python tile expiration
> code.
> 
> There is not other mecanism to re-render tiles globally ?

Not that I'm aware of.

> > > On OSM-FR server, we face some saturation on the tile regeneration process
> > > and in order to find what to improve first, I'm investigating each piece of
> > > the puzzle, from postgres setup, SQL queries, style sheet, cache policy,
> > > expiration, etc...
> > > 
> > > I've done a lot of statistics to understand the ratio of tiles rendered,
> > > cached, requested, purgeable. By analyzing the renderd log, I found that
> > > some tiles get rendered up to 25 times a day !
> > I suspect that people are just busy editing. So there is probably not much
> > you can do there except expiring less often.
> 
> What surprised me is that despite our renderd saturation it did render the
> same tile 25 times in one day !

With minutely updates you also get one expiry run per minute. This basically
means that the tile is invalidated every time somebody does an edit. If you
get a request for the tile in between the edit, you have a rerender on your
hands. 25 times/day corresponds to about 3 edits an hour in a 8-hour work
day. That sounds about right.

Are these 25 renders/day with high zoom tiles or lower zoom?

> Expiring less often is not that easy, re-rendering less often could be done
> by looking at the tile creation timestamp (mtime + 20 years). If the tile is
> expired but not that old, it may be considered not to be rendered in
> priority by mod_tile and provided as is to the client.

The problem here is that you have two kinds of users for your tiles: the
ones that really want to see the changes from a minute ago (aka mappers)
and the ones that would happily look at last weeks tiles without (aka
everybody else). Not rerendering will annoy the first group. Before playing
with expiry, I'd look into what you send as HTTP cache control headers first.
Sending longer expiry to clients might reduce the number of requests from
the second group while the first group still can hit refresh and get their
minutely updated tiles.

> > Both expiry methods still work on raw change data. They do not take into
> > account if the data that has been changed actually has an effect on the
> > rendering. osm2pgsql could in theory do an expiry based on the rendering
> > tables but that would require a major restructuring because it doesn't
> > have a notion of data change yet. Every change is a delete+add operation.
> 
> A "perfect" expiration method would have to be based on the style sheet. It
> is complex but can be worth the effort.
> 
> I'm thinking of intrumenting renderd to compare tiles after a re-rendering
> to get statistics of re-rendered tiles that are the same as the expired ones
> to see the percentage of useless re-rendering.

This would definitely be intersting to have data on. I suspect that the
numbers are very different for different zoom levels. So you might want to
have separate statistics there.

Sarah