[OSM-dev] robots.txt

[Paul Johnson]

On Sun, 15 Aug 2010 21:12:52 +0200, Nic Roets wrote:

> AFAIK, robots.txt only applies to recursive downloads. Given that file
> names follow simple patterns and timestamp files exist, it is really not
> necessary to run recursive spiders. That said, wget and curl can be told
> to ignore robots.txt.

robots.txt can selectively allow wget and curl as well; this would be the 
cleaner solution.