[Tile-serving] [osm2pgsql-dev/osm2pgsql] osm2pgsql using too much memory for expire lists (Issue #2190)

[Jochen Topf]

The current code basically uses a `std::unordered_set<int64_t>` for storing to expired tiles. Entries in an unordered set (internally a hash) have a large memory overhead. We probably need something like 32 bytes for each entry. Looking at the spikes in the graph above it seems we needed something like 64 GB in those cases, which means we had 2 billion tiles in our expire list.

Written out Tom said we had 60 GB of expire list, the list uses lines in a format like `zoom/x/y`. On zoom 18 the largest tile coordinates needs 6 digits each for x and y plus two digits for zoom level plus two `/` characters and a new line, so that's a maximum of 6+6+2+2+1 = 17 characters per entry. Depending on configration the file will also have entries for lower zoom levels. Upper limit for that would be double the size, so 34 characters per entry which puts fits with the calculation above.

2 billion tiles (on z18) is much more than I would ever have expected to see in an expire list. the maximum you can have at z18 is `4^18` which is 64 billion tiles.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/issues/2190#issuecomment-2137067640
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/issues/2190/2137067640 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20240529/f82c411e/attachment.htm>