[Tile-serving] [osm2pgsql-dev/osm2pgsql] osm2pgsql using too much memory for expire lists (Issue #2190)

[Jochen Topf]

One option would be to check the size of the expire list and exit osm2pgsql with an error message if it becomes too large. This is better than just using more and more memory until we are killed by OOM. In addition it would, in some sense, protect later processing steps in the pipeline from overload. Even if we could handle a huge expire list, other parts of the system will not, so it might be better to die early.

This is, of course, not a real solution, because it still means that sysadmin intervention is necessary. But it is something we can implement very easily.

So the idea would be to keep a count of the number of entries in the expire list (or lists) and die if this becomes to large. We'd need to set a maximum number of entries. This should ideally be something that is large enough so that in does not happen in normal day to day operation. We might want to make this configurable, in that case we still need a good default, and some guidance for the sysadmin on how to set this.

We can set this either based on the absolute number of entires in the expire list, a percentage of tiles in the specified zoom level of the expire list, or the memory used (and then use some estimate on how much memory an entry in the expire list needs). I am not sure which value would be best for admins to understand.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2190#issuecomment-2171688389
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2190/2171688389 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20240616/04ca7e4c/attachment.htm>