[OSM-dev] Paging the tile server admins

Kai Krueger kakrueger at gmail.com
Wed Apr 27 16:17:00 BST 2011


NopMap wrote:
> 
> A few days ago a new mass downloading application has appeared on my
> server, already causing tile requests in the order of millions with up to
> 600000 tiles for a single client.
> 

Mod_tile has some built-in QoS / throttling features, that might help
alleviate the effects of large tile scrapers.

There are three parameters in mod_tile.conf with which to control these.

ModTileEnableTileThrottling Off
ModTileThrottlingTiles 10000 1
ModTileThrottlingRenders 128 0.2

It throttles tile downloading per IP with a leaky bucket algorithm (with an
initial bucket size in tiles and a tiles/second rate limit), which should
catch applications even if they fake the UA string.

Apart from tile serving, it also allows to throttle rendering requests at a
different rate. I.e. every time mod_tile can't serve a tile from disk and
has to ask renderd to render it, it will account it into a separate bucket.
This should be particularly effective for low zoom tile scrappers and even
out the use of the rendering queue to be more fair.

The code has support for white listing certain IPs that shouldn't get
throttled, e.g. in case the tile server is behind a proxy server, but there
is no way to configure the white list yet other than in the code requiring a
recompile.

The throttling code also doesn't currently handle IPv6 though.

I haven't heard anyone using these features in mod_tile yet, so although it
should work I am not sure how well it is tested under production yet.
Tile.openstreetmap.org uses a different (squid based) throttling mechanism,
but the built-in way might be easier for smaller tile server installations.

Kai





--
View this message in context: http://gis.638310.n2.nabble.com/Paging-the-tile-server-admins-tp6308590p6309852.html
Sent from the Developer Discussion mailing list archive at Nabble.com.



More information about the dev mailing list