[OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?

Roland Olbricht roland.olbricht at gmx.de
Sun Oct 6 18:15:34 UTC 2013


> In the past I could generally get the query to complete by running it once,
> cancelling it, then running it a second time (maybe the 2nd time more data
> was cached).  Now I get either infinite timeout or a server rejection.
> 
> The query is:
> 
> <osm-script timeout="990" element-limit="1073741824">
> <query type="node">
>   <has-kv k="amenity" v="drinking_water"/>
> </query>
> <print/>
> </osm-script>

I'm glad that you bring up the whole subject. It's a challenging query for 
Overpass API. Malcolm has already sketched the long-term solution: operating 
from a SSD will largely solve the performance limitations, but currently I'm 
not aware of any cost-effective hosting options with large SSDs (more than 300 
GB).

Please do not use a cancel-restart strategy. This may double the load because 
not in all cases the Apache server cancels the abandoned query. It is possible 
that this strategy causes the rejection because the server rejects concurrent 
requests from the same IP.

In fact, if a query is already running a second query from the same IP then 
the new request is first postponed for up to 15 seconds. It is immediately 
started if the former request completes during this time. If the former 
request doesn't complete within 15 seconds then the newer request is finally 
rejected.

I suggest the following strategy: Please prefer the Rambler instance. It is 
for this use case probably the fastest. Then, feel free to use a much larger 
timeout. On a test run, the query had taken 12 minutes, so with a safety 
margin factor, please try 3600 seconds instead of 990. On the other hand, 
please omit the element-limit (and thus use the default of 512M instead). 
That's much more than necessary and the server is rather picky on RAM 
consuming requests.

If the request is still rejected, please retry every few minutes. Peaks most 
often only last for less than an hour but aren't that regular. And trying a 
request doesn't hurt anybody.

Cheers,

Roland




More information about the talk mailing list