[Tile-serving] [openstreetmap/mod_tile] mod_tile appears to require a single-process (but can handle multiple threads) Apache server to work correctly (#238)

[Antti S. Lankila]

I was struggling for some days with the problem that under heavy load, mod_tile seemed to not return a tile promptly to waiting http client connection. The tile was in fact rendered, but the http connection appeared to be stuck waiting for acknowledgement from render, I think.

I read the code a bit and came across this comment in mod_tile/includes/protocol.h:

    A client may not bother waiting for a response if the render daemon is too slow
    causing responses to get slightly out of step with requests.

This gave me cause to investigate the possibility that in a multiprocessing apache, multiple independent http server processes have reached renderd and are writing commands to render tiles and read for response. My hypothesis is that there is a single socket shared by all connected renderd clients, and in that case, if the server that did not submit the request to render a tile reads the response, then it simply throws it away as it can't notify another process's thread, and the Apache process whose thread did submit that request gets stuck waiting for it until something timeouts.

I have no conclusively proven that this is indeed what is happening, but I got rid of my gray, slowly rendered tiles by enabling mod_event and setting it up in such a way that it starts only 1 process, and ThreadLimit, MaxRequests, etc. were all set to 150 so that apache will not fork more than 1 render process. I presume that as long as there is only 1 apache process, when it serves multiple threads, it is able to discover which thread is waiting for the tile rendering acknowledgement.

It is also worth noting that my mod_tile configuration is as follows:

    ModTileTileDir /var/cache/renderd/tiles
    LoadTileConfigFile /etc/renderd.conf
    ModTileEnableStats On
    ModTileRequestTImeout 60
    ModTileMissingRequestTimeout 60
    ModTileRenderdSocketName /run/renderd/renderd.sock

i.e. I have set a long RequestTimeout and MissingRequestTimeout. My clients are willing to wait for as long as it takes for the tile to come, instead of a shorter duration. It may be that the shorter 3 and 10 second timeout values somewhat mask the problem. I think that request does return with the tile's data if the tile gets rendered by renderd before the timeout, it just takes unnecessarily long time to get the tile. In my case, the timeout was excessively long and thus I was motivated to try to figure out why it kept happening.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/mod_tile/issues/238
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210604/a8449378/attachment.htm>