[OSM-talk] Question about Osmosis, Geofabrik and HTTP 504 Error

Juan Lucas Domínguez Rubio juan_lucas_dr at yahoo.com
Wed May 21 16:48:00 UTC 2014


Hello,
Thanks for the explanation.
I have set up the cron so that it gets the *.osc.gz files from Geofabrik every night. I thought it would be a good idea to run osmosis three times, so in case the first does not work, the second perhaps will, and if one of them works, then the rest will do nothing because I am checking the lag every time.

After your reply, I think I will try it only once every night, and if one night it does not work, it's not so important because the following night it will probably work.


Any comments?

Regards,
Juan

On Wednesday, May 21, 2014 5:46 PM, Frederik Ramm <frederik at remote.org> wrote:
 


Hi,


On 05/21/2014 04:36 PM, Juan Lucas Domínguez Rubio wrote:
> I am getting the HTTP 504 error message *sometimes* when trying to get a
> state.txt file from Geofabrik with osmosis, but it happens sometimes and
> with some files. For example, the state.txt file for the Berlin updates
> fails to download and the state.txt file for the Bretagne updates works
> ok, and after a few minutes, the opposite situation happens.
> 
> Could this be because the server is denying files by using some kind of
> file-oriented policy?

The Geofabrik download server applies rate limiting. If you try to
download too many files in a short time - for example, if you were to
run Osmosis in a loop where you'd check for a new state.txt every minute
even though they only are renewed once a day - then the server will
block access for your IP and issue "503 Service Unavailable" errors (but
not 504). Because of the throttling nature of this block, you might land
in a situation where you can request one file every five minutes
successfully, but then you get 503 again.

Don't request so many files, and the problem will go away. If you check
for 10 state.txt files every hour you won't have a problem but if you do
that every five minutes then you might.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"

_______________________________________________
talk mailing list
talk at openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/talk/attachments/20140521/ddf2ce5f/attachment-0001.html>


More information about the talk mailing list