[Tile-serving] [osm2pgsql-dev/osm2pgsql] osm2pgsql using too much memory for expire lists (Issue #2190)

[Jochen Topf]

We should find a better datastructure for keeping the tiles. The current solution is okay for small lists, but if the list gets full, we could switch to a bitmap which gives us a solid upper bound for the memory use. Problem is that that's still too large, especially if we have configured several expire outputs. For z19 we need `4^19 / 8` bytes, thats 32 GB. We could use a bloom filter to reduce memory use but get a certain false positive rate.

A better approach might be to reduce the zoom level dynamically if there are many tiles. With each zoom level we go up we need 4x less memory, so for z17 we are already at a maximum of 2GB which should be okay. The advantage of that approach is that tiles are probably somewhat clustered anyway, so we don't have as many false positives as with the bloom filter. And because we are expering up the zoom levels anyway, the middle zoom levels (which are the expensive ones to render) are unaffected.

Of course the disadvantage with that approach is that while we are protecing osm2pgsql from needing too much memory, we put a somewhat heavier burden on the rendering system.

We can come up with other ways to store the tiles, of course, they probably all of some "failure case", some tile pattern where they need a lot of memory. Something like a checkerboard pattern will probably overstretch all data layouts.

We might also want to look at the way we are reporting the expire list. Currently it is this text file in a CSV-like format. That has, of course, a huge overhead. Any other format needs to be implemented on the consumer side, too. Another option already available for the flex output is storing the expire list in the database.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2190#issuecomment-2141597336
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2190/2141597336 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20240531/80f4212d/attachment.htm>