[Tile-serving] [openstreetmap/mod_tile] Examine memory usage for potential leaks (Issue #445)

[Hummeltech]

I just thought I'd mention my findings on this issue:

* I ran a leak detection and found a [tiny, insignificant leak ](https://github.com/openstreetmap/mod_tile/commit/c9dbcd74c7398f5e1422f3171580e9c2f2e5ae9f)during the configuration parsing. It's 32 bytes and happens once during process startup.

* I [switched out the memory allocator](https://github.com/openstreetmap/mod_tile/commit/dc16ade543d6c358b7ba79cbb19ebc5d350bac3f) (from glib to [jemalloc](http://jemalloc.net)) and that shows promise...

Background: my setup has `renderd` running on a 32GB node with the psqldb sitting on a different node. The database has only a recent North America extract loaded and renders OpenStreetMap-styled tiles with scale=2 (... so they look good on a retina display). Pre-rendering with 8 threads will have the `renderd` processes /threads grow to 31.5GB as soon as zoom levels 7 or higher are reached.

Recompiling with jemalloc yields the following: 0.5GB - 4.7GB per process size at zoom levels 0..11. Right now they are hovering at 1.7GB and churning "ocean tiles" (so little to no features to render)...

My hypothesis is as follows: To render a "busy" tile (one that has a lot of detail at a medium/high zoom level), `renderd.mapnik` requests a *lot* of data from the database. This data is stored in fine-grained allocations which lead to heavy fragmentation of the memory. Although all memory seems to be freed once a tile has been rendered (`mapnik` seems to be releasing everything it allocates), it leaves behind a rather "coarse landscape" and the memory allocations required for the next tile may not fit nicely inside these holes. This causes the standard allocator to just grab some more "fresh" memory from the system (by [increasing the data segment ceiling](https://en.wikipedia.org/wiki/Sbrk)), which results in a bigger and bigger process memory usage. If the memory allocator were to coalesce adjacent holes more aggressively, it might be able to find entire "pages" to be returned to the system, but I don't know if the glib allocator has this feature at all. However, `jemalloc` can do that.

Fragmentation is still an issue (and will always be with a non-garbage collecting system/language), but switching to this new allocator has shown a much nicer memory consumption pattern.

PS: My leak detection findings are very rudimentary and I don't claim that I found all leaks. But to further investigate, `renderd` should be able to terminate gracefully and shut down all threads. As it is coded right now, it just runs "forever"...

_Originally posted by @rolandbosa in https://github.com/openstreetmap/mod_tile/issues/181#issuecomment-2164503429_
            

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/mod_tile/issues/445
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/mod_tile/issues/445 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20240614/185023ef/attachment.htm>