[Tile-serving] [openstreetmap/osm2pgsql] Tile Expiry Performance (#946)

[Sarah Hoffmann]

I'm not surprised. The expired tiles are saved in a set with 8 bytes per tile. Each parallel process keeps its own tile list which are merged only in the end. So that's in your case 4 * 8 bytes in the worst case. Add overhead for the hash and there is quite a bit of memory required for high zoom levels and large changesets. There is some room for improvement wrt the data structures used. However, there is also some cut-off point where I would consider the use of expiry unrealistic. Applying a weekly diff and creating expiry list for z19 is one of them. I wouldn't expect the expiry to be optimised for that. A daily diff at z16 should be doable though.

@Nakaner I trust you look at some point into optimising the data structures a bit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/946#issuecomment-521419817
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190814/cdff7ebd/attachment.html>