[OSM-talk] Overpass Quotas

Tue May 5 20:21:49 UTC 2015

Dear all,

the last mail needs some more clarification. While I appreaciate that 
quite a lot of people care for not overusing the server, they are not 
the ones I have meant with "clumsy clients".

I fear it is quite a common problem and probably has a name: those who 
care about usage policies restrict themselves even more, but it doesn't 
help, because the problem comes from people that don't care or don't 
understand that they cause trouble.

In other words: if you read this message then you don't contribute to 
congestion.

Hence, if you have never seen a "HTTP 429 Too much requests" message, 
then your usage of overpass-api.de is perfectly fine. Please carry on 
and don't mind.

Even if you have seen every now and then (as a hint: in less than 20 
percent of all requests) a HTTP 429 then just relax a little bit, try to 
move long running requests (beyond 180 seconds, if any) off peak times 
(the time from UTC 12:00 to UTC 22:00 is more busy than the rest). And 
still everything is fine, even if you cannot move out of peak times. The 
server will not keep any record after rejection, and you will be treated 
after a few minutes as if you never have been involved in a HTTP 429 
request.

You should be only concerned if you find yourself on this short list of 
blocked IP addresses:

5.9.65.X
104.47.142.X
104.40.194.X
167.114.116.0/22
58.51.144.X
58.51.144.Y
54.78.255.X
54.247.44.X

The next risk candidates are:

188.166.8.X
194.33.69.X

A general misbehaving group of clients has been identified by sending 
the user request "Only a test!". While I appreciate to send a 
self-chosen user agent, I have difficulties to derive a point of contact 
from that specific user agent.

Once again: In all other cases, your usage of overpass-api.de is 
responsible and perfectly fine.

For the curious, some more data:

01/Apr/2015     111710  67894   700976  488641  38386
02/Apr/2015     125437  73602.1 774208  506222  36288
03/Apr/2015     111768  66832.3 821650  495482  35969
04/Apr/2015     119659  49824.3 906004  459878  32267
05/Apr/2015     106234  45427.5 881590  462305  33588
06/Apr/2015     83007.5 43324.7 930086  505258  32662
07/Apr/2015     100788  42118.3 936754  511573  33880
08/Apr/2015     87590.3 40828.3 788823  474827  33610
09/Apr/2015     112620  37723.3 663166  450491  31912
10/Apr/2015     114220  37376.3 665173  465647  30522
11/Apr/2015     69314.6 26770   661636  439874  28347
12/Apr/2015     113752  36699.7 631125  469711  30236
13/Apr/2015     67609.4 30192.6 742857  475160  33531
14/Apr/2015     47800.8 26262.2 732373  456884  31838
15/Apr/2015     86428.3 30434.3 733738  484012  32638
16/Apr/2015     94440.9 28273.7 783472  466555  32041
17/Apr/2015     74105.2 26220   829937  471074  30380
18/Apr/2015     63287.7 16449.5 851242  460553  27105
19/Apr/2015     87040.5 20821.3 775956  471591  30032
20/Apr/2015     71236.3 28174.4 775381  471193  31782
21/Apr/2015     74426.7 27312.3 805662  423813  30961
22/Apr/2015     76482.8 32474.2 776979  458964  30663
23/Apr/2015     63931   30986   779185  400018  29209
24/Apr/2015     65186.2 29074.9 713110  414088  30181
25/Apr/2015     63562   20196.7 604212  355809  28733
26/Apr/2015     61562.2 22193.2 600918  362236  29523
27/Apr/2015     95413.6 33839.2 627806  407742  32568
28/Apr/2015     71292.1 30761   678194  411622  32213
29/Apr/2015     108186  29082.4 692393  416049  30986
30/Apr/2015     126114  28901.1 857178  397983  29670

The second column is the total amount of data in megabytes by day. The 
third column is the total amount of data in megabytes per day for all 
except the 30 heaviest users. You can notice that on most days these 30 
heaviest users consume 50% to 80% of all data.

In a similar manner, the fourth and fifth column list the number of 
requests. Again, heavy users are quite dominant.

The last column is the number of unique IP addresses the server has seen 
that day.

The full diagram has more columns to tell apart the heaviest user, the 3 
heaviest users, and the 10 heaviest users. However, the tendence remains 
the same that very few (less than 1 in 1000 users) consume more than 50% 
of the resources.

So if you are not in this top-30-list, then your usage of 
overpass-api.de is perfectly fine.

Happy querying,

Roland