[OSM-talk] Overpass or XAPI API servers for 60, 000 node queries?
roland.olbricht at gmx.de
Sun Oct 6 18:15:34 UTC 2013
> In the past I could generally get the query to complete by running it once,
> cancelling it, then running it a second time (maybe the 2nd time more data
> was cached). Now I get either infinite timeout or a server rejection.
> The query is:
> <osm-script timeout="990" element-limit="1073741824">
> <query type="node">
> <has-kv k="amenity" v="drinking_water"/>
I'm glad that you bring up the whole subject. It's a challenging query for
Overpass API. Malcolm has already sketched the long-term solution: operating
from a SSD will largely solve the performance limitations, but currently I'm
not aware of any cost-effective hosting options with large SSDs (more than 300
Please do not use a cancel-restart strategy. This may double the load because
not in all cases the Apache server cancels the abandoned query. It is possible
that this strategy causes the rejection because the server rejects concurrent
requests from the same IP.
In fact, if a query is already running a second query from the same IP then
the new request is first postponed for up to 15 seconds. It is immediately
started if the former request completes during this time. If the former
request doesn't complete within 15 seconds then the newer request is finally
I suggest the following strategy: Please prefer the Rambler instance. It is
for this use case probably the fastest. Then, feel free to use a much larger
timeout. On a test run, the query had taken 12 minutes, so with a safety
margin factor, please try 3600 seconds instead of 990. On the other hand,
please omit the element-limit (and thus use the default of 512M instead).
That's much more than necessary and the server is rather picky on RAM
If the request is still rejected, please retry every few minutes. Peaks most
often only last for less than an hour but aren't that regular. And trying a
request doesn't hurt anybody.
More information about the talk