[Osmf-talk] February OSMF Board Meeting

[Tom Hughes]

On 26/02/10 10:17, Gervase Markham wrote:

> Oh, and can we take the front page out of robots.txt?
> http://web.archive.org/web/*/http://www.openstreetmap.org

Probably, if you can tell me how. Unfortunately the robots.txt standard 
is so completely shit that in the end I gave up trying to block the 
things that were causing problems and just blocked everything in favour 
of using a sitemap instead.

If you can give me robots.txt speak that allows the home page and blocks 
everything else then we could probably use that.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://compton.nu/