[openstreetmap-website] Add Host: to robots.txt (#1117)

Grant notifications at github.com
Sat Dec 26 23:56:41 UTC 2015

Add a Host: which is an extension to the robots.txt standard. https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host

Googlebot particularly seems to be incorrectly indexing the spike-0x host urls.
You can view, comment on, or merge this pull request online at:


-- Commit Summary --

  * Add Host: to robots.txt

-- File Changes --

    M public/robots.txt (2)

-- Patch Links --


Reply to this email directly or view it on GitHub:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20151226/629b3b94/attachment.html>

More information about the rails-dev mailing list