[OSM-dev] tips to reduce bulk tiles downloads ?

[sly (sylvain letuffe)]

On mercredi 27 juillet 2011, NopMap wrote:
> You know about the throttling of download speed in mod_tile, I guess.

Well no, because I don't use mod_tile, but that is a good idea, instead of 
blocking completly the ability to download, reducing/limiting the hit rate 
per minute per IP might be a good idea. I'll try to do something about that.

> On my server I use a tile limit per zoom level (automated tile scrapers show
> an exponential increase with higher zoom levels) 

I plan to add a "tile not in the cache penalty" because that's what put all 
the load, deliverying tiles in cache as close to no impact

> and a penalty for 
> missing/fake agents. 

How do you know what are fake agents ?

> Excessive IPs are blocked for 24h.  
That's what I'm currently doing, but that's not perfect

> Actually, a good strategy is to try and identify the application causing the
> load and talk to the author. 
That sound a good way to find compromise, I don't want to ban, I just want my 
server not to be sluggish like snails for legitimate usage, but I have 
problem identifying applications :

> Many applications do have a user agent. 
> Currently the most frequently blocked downloaders in my log are MOBAC and
> Locus.
> 
> Do you have any idea who is causing the load?

By far the one with more hits (before I rate limited hits) looks strange :
It's refererer is :
file:///Applications/Install/83F78CDD-FB29-E011-854C-00237DE2DB9E/Install/
and user agent is : NativeHost

The second one has what looks a fake user-agent :
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)"
(sequential download and IE 6.0, seriously ;-) )

Then comes (or alike):
"Dalvik/1.2.0 (Linux; U; Android 2.2.2; LS670 Build/FRG83G)"

<troll>But android apps seams more friendly with my server as I didn't find 
what looks bulk dowloads. android devs more respectfull ? ;-) </troll>

I don't know what MOBAC is but I only found 1000 hits 
with : "MOBAC/1.9_beta_6" in 6 days, so that's clearly not a problem for me.

-- 
sly
qui suis-je : http://sly.letuffe.org