[Tile-serving] High renderd memory consumption ?

[Kai Krueger]

On 06/15/2013 02:58 AM, sly (sylvain letuffe) wrote:
> Le vendredi 14 juin 2013 21:52:16, sly (sylvain letuffe) a écrit :
>
>> I'll collect more info and open an issue on github, for now, the
>> informations I have are rather weak.
> Weak information... and changing...
> I compiled mod_til+renderd from ~10 hours ago and let that run since then and
> no more important Memory consumption happened (while on my setup it only
> needed like an hour to show the memory exhaustion before)
>
> Did that commit solved it :
> https://github.com/openstreetmap/mod_tile/commit/dd75339f8d4144d1bc31370c599e5b1bb79b10b0
> ?
No, it is unlikely that this commit solved it. Currently renderd runs in 
an infinite loop and never shutsdown or frees the long running memory, 
so it doesn't close the queues either. Once the process gets killed, the 
memory gets released anyway. However, that does make it difficult to run 
valgrind on renderd. This commit was therefore mainly for the test 
framework to try and make it memory leak free, so that I can run 
valgrind on the unit test framework instead.
> I couldn't be sure since I haven't wiped out my metatile store, and some
> longer to generate tile might have been the cause.
For some of the low zoom tiles, I think mapnik (and therefore renderd) 
can use several hundered MB of ram for rendering. If you then have many 
threads each rendering low zoom, I guess it can add up into the GB range 
of ram used. However, once the tile has finished rendering, that memory 
should be freed and the memory usage of renderd should be back down to 
the amount with which it started up.

Unless you are running on Solaris. It's malloc implementation is imho 
totally broken for server processes, as it never frees memory back to 
the OS. So a process always uses as much memory as during its peak, or 
possibly even grow.
>
> I'll wait a little more and report if this behaviour is still to me seen.

Thanks.
>
>
>