[OSM-dev] mod_tile memory leak?

[Jon Burgess]

On Tue, 2008-06-03 at 22:48 -0500, Dylan Semler wrote:
> Without knowing exactly what's going on, it seems that I think
> mod_tile has a memory problem.  IANAP, but after rendering a tile and
> converting the .meta, it seems all of the data stays in memory.
> Rendering subsequent tiles builds up the memory footprint quite
> quickly--at z5 and 4G of ram I can only generate 1 meta file at a time
> without restarting renderd to clear the memory.
> 
> For example:
> 
> Rendering daemon
> Got incoming connection, fd 5, number 1
> Render fd(5) z(5), x(4), y(12)
> Got incoming connection, fd 9, number 2
> Render fd(9) z(5), x(5), y(12)
> Got incoming connection, fd 10, number 3
> Got incoming connection, fd 11, number 4
> ...
> Render fd(14) z(5), x(2), y(8)
> Render fd(7) z(5), x(1), y(8)      <----  Up to here memory remains
> reasonable, maybe  20% used by "programs", 80% "cache"
> DONE TILE 5 0-7 8-15             <----  Here memory starts building up
> until the .meta file is complete.  Here 80% "programs", 20% "cache"
> Produced .meta: /var/www/html/direct/5/0/0/0/0/8.meta
> 
> At this point the .meta file is generated but RAM usage does not drop.
> Killing renderd causes RAM usage to drop back to normal (10%
> "programs", 20% "cache").
> 
> Perhaps obviously I'm using "System Monitor" in gnome-panel to measure
> memory usage.  Let me know if something more scientific is desired.


The system monitor is a little crude. Most linux systems that have been
running for any reasonable length of time will show nearly zero free
memory. This is expected behaviour[1][2].

You suggest that you have to restart the renderd. If you don't do this,
do you see anything bad happening?

The code is used on the live OSM site rendering millions of tiles per
day. It can use a fair amount of memory when it is running but I don't
see it increasing over time (which would signify a leak). 

If you want to cut down the memory usage then you can make it single
threaded: edit render_config.h to set NUM_THREADS to 1, then rebuild
renderd.

	Jon


[1] http://techtips.chanduonline.com/2006/08/11/linux-uses-too-much-memory-a-very-basic-linux-memory-guide/
[2] http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html