[Tile-serving] [openstreetmap/osm2pgsql] Rewrite tile expiry (management of expired tiles) (#709)

[Michael Reichert]

@zerebubuth wrote:
> I notice that the blue tiles appear to be twice as large as the red tiles, suggesting that they're at one zoom level lower. Is that the case? Has the output changed if only a single zoom level is requested (e.g: --expire-tiles 16-16)?

The old tree implementation silently drops some tiles (the blue ones) at any point in the code. I don't know why it does so. The unordered set does not drop anything.

Dropping the tiles is not documented, neither in the user documentation nor in the source code. I don't know if it is a bug or a feature. I myself expected the tile expiry to behave as my implementation does.

> If the two methods are not (almost) the same for outputting just a single zoom, then I think I'd want to understand the difference in the selection of tiles (more tiles, fewer tiles, by what proportion?) and the reason why before I updated to the new expiry algorithm.
> 
> Not that a change is necessarily bad, and I don't think it should hold up this issue being resolved. I think it would be really useful to have a thorough description of what's changed and what differences users can expect in the changelog / release notes.

Pull request #747 does not change the selection of the tiles based on the geometry of the changed features in the database. That is still done by the legacy code (`expire_tiles::from_line(double, double, double, double)`, `expire_tiles::from_bbox(double, double, double, double)`). These methods call `expire_tiles::expire_tile(uint32_t, uint32_t)` (formerly `expire_tiles::expire_tile(int, int)`) which adds the tile to the set. The PR changed `expire_tiles::expire_tile(uint32_t, uint32_t)` and all methods which are called by this method.

Following text could be used for the release notes:

> **Rewrite of the Tile Expiry**
> 
> The tile expiry now uses internally an unorderd set instead of a self-implemented tree. It now outputs *all* tiles on all requested zoom levels which contain a map change.
> 
> During the rewrite a bug in the old code was found (and fixed) which dropped about two thirds of the expired tiles. If you used the old tile expiry for a single zoom level, the list of expired tiles will increase by the factor 2 to 3.
> 
> If you used the old tile expiry for multiple zoom levels (e.g. z..z+n), you will now get all tiles which contain changes. The old implementation summarized the list in following manner: A tile on zoom level z was only written to the output file if all sub-tiles on zoom levels z+1..z+n contained a map change. The new implementation will output the tile on zoom level z even if only one sub-tile contains a map change. The following image illustrates the changes.
> 
> ![changes of osm2pgsql illustrated](https://michreichert.de/forum-images/github/openstreetmap/osm2pgsql/osm2pgsql-expiry-change.svg)

license of the text and the image: CC-0

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/709#issuecomment-302674031
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20170519/923d171a/attachment-0001.html>