[Tile-serving] [openstreetmap/mod_tile] Time spent per tile changes to 4.2 seconds when there is a queue (Issue #333)

[Paul Norman]

The OSMF got a new tile server, and in scaling it up, I've observed some odd behavior.

When I throw a heavy load onto the server, a queue builds up. This is semi-expected, but the average time per tile, as computed by `TimeRenderedZoom/ZoomRendered` becomes approximately 4.2 seconds. Normally it ranges from <1s for high zooms to 5-10s for zoom 13.

This can be seen in [prometheus](https://prometheus.openstreetmap.org/d/wyyzhZKMk/tile-rendering?orgId=1&from=1694715282056&to=1694716812056&var-instance=piasa)

![image](https://github.com/openstreetmap/mod_tile/assets/1190866/dd46f608-8361-491d-bc2b-912549f47969)

With a bit of work I was able to see this with a [partially filled queue](https://prometheus.openstreetmap.org/d/wyyzhZKMk/tile-rendering?orgId=1&from=1694720569970&to=1694721594263&var-instance=piasa)
 although in this case it's just the high zooms. 

![image](https://github.com/openstreetmap/mod_tile/assets/1190866/2437898e-c33c-4814-9bc3-95019c2d5fd6)

Looking at the code that produces the render time and number of tiles stats, they're set in https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/request_queue.c#L371-L401 with render_time coming from https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/gen_tile.cpp#L505, which is essentially the call time of `maps[i].store->tile_stat` and `render`, in ms.

I'm somewhat stumped here, but a few thoughts are
- this is limiting render capacity well below what it should be
- the machine has idle capacity throughout, particularly CPU
- why 4.2s?
- how does the average zoom 13 time per tile *decrease*?

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/mod_tile/issues/333
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/mod_tile/issues/333 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20230914/638342c0/attachment.htm>