[OSM-talk] Question about Osmosis, Geofabrik and HTTP 504 Error

[Frederik Ramm]

Hi,

On 05/21/2014 04:36 PM, Juan Lucas Domínguez Rubio wrote:
> I am getting the HTTP 504 error message *sometimes* when trying to get a
> state.txt file from Geofabrik with osmosis, but it happens sometimes and
> with some files. For example, the state.txt file for the Berlin updates
> fails to download and the state.txt file for the Bretagne updates works
> ok, and after a few minutes, the opposite situation happens.
> 
> Could this be because the server is denying files by using some kind of
> file-oriented policy?

The Geofabrik download server applies rate limiting. If you try to
download too many files in a short time - for example, if you were to
run Osmosis in a loop where you'd check for a new state.txt every minute
even though they only are renewed once a day - then the server will
block access for your IP and issue "503 Service Unavailable" errors (but
not 504). Because of the throttling nature of this block, you might land
in a situation where you can request one file every five minutes
successfully, but then you get 503 again.

Don't request so many files, and the problem will go away. If you check
for 10 state.txt files every hour you won't have a problem but if you do
that every five minutes then you might.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"