Thanks for your response!<div><br></div><div>My output from the daemon log doesn't show anything about threads crashing.</div><div><br></div><div><div><font face="courier new, monospace" size="1">Jun 19 21:05:24 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 7992 (due to timeout) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:05:40 maptile tirex-backend-manager[7840]: sending TERM to worker 'mapnik' with pid 7992 (due to timeout) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:05:40 maptile tirex-backend-manager[7840]: child 7992 terminated (exit_code=0, signal=15) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:05:40 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15666 </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:05:41 maptile tirex-backend-mapnik[15666]: Renderer started (name=mapnik)</font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:39 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 9375 (due to timeout) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:39 maptile tirex-backend-manager[7840]: child 9375 terminated (exit_code=127, signal=0) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:39 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15683 </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:39 maptile tirex-backend-mapnik[15683]: Renderer started (name=mapnik)</font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:44 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 7991 (due to timeout) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:44 maptile tirex-backend-manager[7840]: child 7991 terminated (exit_code=127, signal=0) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:44 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15694 </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:44 maptile tirex-backend-mapnik[15694]: Renderer started (name=mapnik)</font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:46 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 9523 (due to timeout) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:46 maptile tirex-backend-manager[7840]: child 9523 terminated (exit_code=127, signal=0) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:46 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15695 </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:46 maptile tirex-backend-mapnik[15695]: Renderer started (name=mapnik)</font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:49 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 7995 (due to timeout) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:49 maptile tirex-backend-manager[7840]: child 7995 terminated (exit_code=127, signal=0) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:49 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15696 </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:07:49 maptile tirex-backend-mapnik[15696]: Renderer started (name=mapnik)</font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:07:57 maptile tirex-master[7852]: Job with id=1340135843_499660176 timed out on rendering list (map=metamaps z=7 x=64 y=40) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:08:09 maptile tirex-backend-manager[7840]: sending HUP to worker 'mapnik' with pid 7994 (due to timeout) </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:08:09 maptile tirex-backend-manager[7840]: child 7994 terminated (exit_code=127, signal=0) </font></div><div><font face="courier new, monospace" size="1">Jun 19 21:08:09 maptile tirex-backend-manager[7840]: renderer mapnik started with pid 15697 </font></div>
<div><font face="courier new, monospace" size="1">Jun 19 21:08:09 maptile tirex-backend-mapnik[15697]: Renderer started (name=mapnik)</font></div><div><br></div><div>Tirex appears to match up with Mapnik.</div><div><br></div>
<div><div><font face="courier new, monospace" size="1">root@maptile:/var/log# tirex-status -o</font></div><div><font face="courier new, monospace" size="1">ps</font></div><div><font face="courier new, monospace" size="1"> Tirex Master Status (updated=2012-06-19 21:55:10)</font></div>
<div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1"> Master server:</font></div><div><font face="courier new, monospace" size="1"> started=2012-06-19 16:07:57 pid=7852</font></div>
<div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1"> Statistics:</font></div><div><font face="courier new, monospace" size="1"> count_error=0</font></div>
<div>
<font face="courier new, monospace" size="1"> count_expired=0</font></div><div><font face="courier new, monospace" size="1"> count_rendered[metamaps]=0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1, 0, 4</font></div><div><font face="courier new, monospace" size="1"> count_requested=12</font></div>
<div><font face="courier new, monospace" size="1"> count_timeouted=4</font></div><div><font face="courier new, monospace" size="1"> sum_render_time[metamaps]=0, 0, 0, 0, 0, 0, 0, 536493, 0, 0, 0, 0, 0, 23375, 0, 60094</font></div>
<div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1"> Queue:</font></div><div><font face="courier new, monospace" size="1"> Prio Size Maxsize Age</font></div>
<div><font face="courier new, monospace" size="1"> 1 0 1</font></div><div><font face="courier new, monospace" size="1"> 2 0 1</font></div><div><font face="courier new, monospace" size="1"> </font></div>
<div><font face="courier new, monospace" size="1"> all 0 1</font></div><div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1"> Buckets: (load=0)</font></div>
<div><font face="courier new, monospace" size="1"> Name Priority Rendering MaxRend Maxload Active Can Queued Age</font></div><div><font face="courier new, monospace" size="1"> live 1- 9 1 19 28 yes yes 0</font></div>
<div><font face="courier new, monospace" size="1"> important 10- 19 0 15 20 yes yes 0</font></div><div><font face="courier new, monospace" size="1"> background 20- 0 4 15 yes yes 0</font></div>
<div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1"> Currently rendering: (num=1)</font></div><div><font face="courier new, monospace" size="1"> Map X Y Z Prio Age</font></div>
<div><font face="courier new, monospace" size="1"> metamaps 64 40 7 1 132</font></div><div><font face="courier new, monospace" size="1"><br></font></div><div><font face="courier new, monospace" size="1">root@maptile:/var/log# ps -ef | grep -i mapnik</font></div>
<div><font face="courier new, monospace" size="1">tirex 7845 7840 0 16:07 ? 00:00:38 mapnik: idle </font></div><div><font face="courier new, monospace" size="1">tirex 7993 7840 0 16:17 ? 00:00:07 mapnik: idle </font></div>
<div><font face="courier new, monospace" size="1">tirex 7996 7840 0 16:17 ? 00:00:07 mapnik: rendering z=7 x=64 y=40 map=metamaps </font></div><div><font face="courier new, monospace" size="1">tirex 15666 7840 0 21:05 ? 00:00:00 mapnik: idle </font></div>
<div><font face="courier new, monospace" size="1">tirex 15683 7840 0 21:07 ? 00:00:00 mapnik: idle </font></div><div><font face="courier new, monospace" size="1">tirex 15694 7840 0 21:07 ? 00:00:00 mapnik: idle </font></div>
<div><font face="courier new, monospace" size="1">tirex 15695 7840 0 21:07 ? 00:00:00 mapnik: idle </font></div><div><font face="courier new, monospace" size="1">tirex 15696 7840 0 21:07 ? 00:00:00 mapnik: idle </font></div>
<div><font face="courier new, monospace" size="1">tirex 15697 7840 0 21:08 ? 00:00:00 mapnik: idle </font></div><div><font face="courier new, monospace" size="1">root 16302 16250 0 21:55 pts/0 00:00:00 grep --color=auto -i mapnik</font></div>
<div><font face="courier new, monospace" size="1">root@maptile:/var/log# </font></div></div><div><br></div>I'm really struggling now. I enabled verbose logging in Tirex, which doesn't seem to have produces any better logging anywhere. Tirex isn't crashing, it just refuses to accept connections while this one tile is in the queue, or until HTTPD times out.</div>
<div><br></div><div>Here's my apache2 error.log</div><div><br></div><div><div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [info] [client 10.0.102.1] Global planet file time stamp (/var/lib/mod_tile//planet-import-complete) updated to Tue, 29 May 2012 12:22:56 GMT, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(864): [client 10.0.102.1] tile_handler_serve: xml(metamaps) z(8) x(129) y(87), referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(325): [client 10.0.102.1] expires(tile_serve), uri(/tiles/metamaps/8/130/87.png), filename(/var/lib/mod_tile/metamaps/8/0/0/0/133/0.meta), path_info((null))\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(363): [client 10.0.102.1] caching heuristics: next planet render -1240360; zoom level based 11104000; last modified 0\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(374): [client 10.0.102.1] Setting tiles maxAge to 11113074\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(325): [client 10.0.102.1] expires(tile_serve), uri(/tiles/metamaps/8/129/87.png), filename(/var/lib/mod_tile/metamaps/8/0/0/0/133/0.meta), path_info((null))\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(363): [client 10.0.102.1] caching heuristics: next planet render -1240360; zoom level based 11104000; last modified 0\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(374): [client 10.0.102.1] Setting tiles maxAge to 11113074\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(325): [client 10.0.102.1] expires(tile_serve), uri(/tiles/metamaps/8/125/87.png), filename(/var/lib/mod_tile/metamaps/8/0/0/0/117/128.meta), path_info((null))\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(363): [client 10.0.102.1] caching heuristics: next planet render -1240360; zoom level based 11104000; last modified 0\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:36 2012] [debug] ./mod_tile.c(374): [client 10.0.102.1] Setting tiles maxAge to 11113074\n, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:42 2012] [debug] ./mod_tile.c(925): [client 127.0.0.1] tile_translate: uri(*)</font></div><div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:42 2012] [debug] ./mod_tile.c(950): [client 127.0.0.1] tile_translate: testing baseuri(/tiles/metamaps/) name(metamaps) extension(png)</font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:42 2012] [debug] ./mod_tile.c(1016): [client 127.0.0.1] tile_translate: No suitable tile layer found</font></div><div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:43 2012] [debug] ./mod_tile.c(925): [client 127.0.0.1] tile_translate: uri(*)</font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:55:43 2012] [debug] ./mod_tile.c(950): [client 127.0.0.1] tile_translate: testing baseuri(/tiles/metamaps/) name(metamaps) extension(png)</font></div><div>
<font size="1" face="courier new, monospace">[Tue Jun 19 21:55:43 2012] [debug] ./mod_tile.c(1016): [client 127.0.0.1] tile_translate: No suitable tile layer found</font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(197): [client 10.0.102.1] request_tile: Request xml(metamaps) z(7) x(67) y(41) could not be rendered in 50 seconds, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(758): [client 10.0.102.1] tile_storage_hook: Missing tile was not rendered in time. Returning File Not Found, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] mod_deflate.c(615): [client 10.0.102.1] Zlib: Compressed 224 to 176 : URL /tiles/metamaps/7/67/41.png, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(197): [client 10.0.102.1] request_tile: Request xml(metamaps) z(7) x(67) y(44) could not be rendered in 50 seconds, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(758): [client 10.0.102.1] tile_storage_hook: Missing tile was not rendered in time. Returning File Not Found, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] mod_deflate.c(615): [client 10.0.102.1] Zlib: Compressed 224 to 176 : URL /tiles/metamaps/7/67/44.png, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(197): [client 10.0.102.1] request_tile: Request xml(metamaps) z(7) x(67) y(42) could not be rendered in 50 seconds, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(758): [client 10.0.102.1] tile_storage_hook: Missing tile was not rendered in time. Returning File Not Found, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] mod_deflate.c(615): [client 10.0.102.1] Zlib: Compressed 224 to 176 : URL /tiles/metamaps/7/67/42.png, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(197): [client 10.0.102.1] request_tile: Request xml(metamaps) z(7) x(67) y(43) could not be rendered in 50 seconds, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(758): [client 10.0.102.1] tile_storage_hook: Missing tile was not rendered in time. Returning File Not Found, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] mod_deflate.c(615): [client 10.0.102.1] Zlib: Compressed 224 to 176 : URL /tiles/metamaps/7/67/43.png, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(197): [client 10.0.102.1] request_tile: Request xml(metamaps) z(7) x(67) y(40) could not be rendered in 50 seconds, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] ./mod_tile.c(758): [client 10.0.102.1] tile_storage_hook: Missing tile was not rendered in time. Returning File Not Found, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
<div><font size="1" face="courier new, monospace">[Tue Jun 19 21:56:23 2012] [debug] mod_deflate.c(615): [client 10.0.102.1] Zlib: Compressed 224 to 176 : URL /tiles/metamaps/7/67/40.png, referer: <a href="http://maptiles.cust.m4.net/">http://maptiles.cust.m4.net/</a></font></div>
</div><div><br></div><div><br></div><div>Any more ideas?</div><div><br></div><div>Thanks again.</div><div><br></div><div><br></div><div><div class="gmail_quote">On 19 June 2012 15:52, Frederik Ramm <span dir="ltr"><<a href="mailto:frederik@remote.org" target="_blank">frederik@remote.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<div class="im"><br>
<br>
On 06/19/2012 03:45 PM, Richard Ive wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In my second screen shot you can see the two tiles that get requested<br>
just before the freeze. I can consistently make this happen and resolve<br>
the hang by restarting Tirex, which would suggest the problem is there,<br>
however looking through all the logs I don't see anything related to<br>
this issue.<br>
</blockquote>
<br></div>
If a Tirex render process dies - something that you would see in the "daemon" syslog - then the effect is that tirex-master still lists the request as being rendered but there will be no "mapnik" process in the system that claims to be rendering this job. If this happens often enough, the Tirex queue will be full. Look at tirex-status and compare the list of tiles being rendered with the list of "mapnik" processes. If processes are indeed dying, activate verbose=true in the Tirex config and try to find out; it might well be a funny Mapnik exception (albeit unlikely that something breaks a whole column of tiles).<br>
<br>
Bye<span class="HOEnZb"><font color="#888888"><br>
Frederik<br>
<br>
-- <br>
Frederik Ramm ## eMail <a href="mailto:frederik@remote.org" target="_blank">frederik@remote.org</a> ## N49°00'09" E008°23'33"<br>
<br>
______________________________<u></u>_________________<br>
dev mailing list<br>
<a href="mailto:dev@openstreetmap.org" target="_blank">dev@openstreetmap.org</a><br>
<a href="http://lists.openstreetmap.org/listinfo/dev" target="_blank">http://lists.openstreetmap.<u></u>org/listinfo/dev</a><br>
</font></span></blockquote></div><br>
</div>