[OSM-dev] robots.txt

Roland Ramthun osm-ml at roland-ramthun.de
Sun Aug 15 20:15:22 BST 2010

On Sun, 15 Aug 2010 12:39:34 -0400
Anthony <osm at inbox.org> wrote:

> I see http://planet.openstreetmap.org/robots.txt now has User-agent: *
> and Disallow: /
> Are we allowed to download the minute-replicate files as they become
> available?  If not, what's the point of having them?

I think the sysadmins just want to block search engine spiders which download bz2 files.

Your own scripts are robots, but I think it isn't considered abusive if they don't obey robots.txt in case they only download the diffs (once) and apply them to a database.

If you want an explicit permission, ask someone from the admin team to create a robots.txt entry in this form:
User-agent: AnthonyBot

User-agent: *
Disallow: /


More information about the dev mailing list