<p></p>
<p dir="auto">The OSMF got a new tile server, and in scaling it up, I've observed some odd behavior.</p>
<p dir="auto">When I throw a heavy load onto the server, a queue builds up. This is semi-expected, but the average time per tile, as computed by <code class="notranslate">TimeRenderedZoom/ZoomRendered</code> becomes approximately 4.2 seconds. Normally it ranges from <1s for high zooms to 5-10s for zoom 13.</p>
<p dir="auto">This can be seen in <a href="https://prometheus.openstreetmap.org/d/wyyzhZKMk/tile-rendering?orgId=1&from=1694715282056&to=1694716812056&var-instance=piasa" rel="nofollow">prometheus</a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/1190866/268095644-dd46f608-8361-491d-bc2b-912549f47969.png"><img src="https://user-images.githubusercontent.com/1190866/268095644-dd46f608-8361-491d-bc2b-912549f47969.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">With a bit of work I was able to see this with a <a href="https://prometheus.openstreetmap.org/d/wyyzhZKMk/tile-rendering?orgId=1&from=1694720569970&to=1694721594263&var-instance=piasa" rel="nofollow">partially filled queue</a><br>
although in this case it's just the high zooms.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/1190866/268096053-2437898e-c33c-4814-9bc3-95019c2d5fd6.png"><img src="https://user-images.githubusercontent.com/1190866/268096053-2437898e-c33c-4814-9bc3-95019c2d5fd6.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">Looking at the code that produces the render time and number of tiles stats, they're set in <a href="https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/request_queue.c#L371-L401">https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/request_queue.c#L371-L401</a> with render_time coming from <a href="https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/gen_tile.cpp#L505">https://github.com/openstreetmap/mod_tile/blob/24a5752d93bb04477bfd96255f0d4f0fbb9e4867/src/gen_tile.cpp#L505</a>, which is essentially the call time of <code class="notranslate">maps[i].store->tile_stat</code> and <code class="notranslate">render</code>, in ms.</p>
<p dir="auto">I'm somewhat stumped here, but a few thoughts are</p>
<ul dir="auto">
<li>this is limiting render capacity well below what it should be</li>
<li>the machine has idle capacity throughout, particularly CPU</li>
<li>why 4.2s?</li>
<li>how does the average zoom 13 time per tile <em>decrease</em>?</li>
</ul>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/openstreetmap/mod_tile/issues/333">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AA6353TUAQ3SLDDSI3OMLPTX2NQZBANCNFSM6AAAAAA4YXYQ6U">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/AA6353RHLIK5PZ3K6RUGO5TX2NQZBA5CNFSM6AAAAAA4YXYQ6WWGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHHCFM7SU.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><openstreetmap/mod_tile/issues/333</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/openstreetmap/mod_tile/issues/333",
"url": "https://github.com/openstreetmap/mod_tile/issues/333",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>