[openstreetmap-website] Add Host: to robots.txt (#1117)

[Grant]

Add a Host: which is an extension to the robots.txt standard. https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host

Googlebot particularly seems to be incorrectly indexing the spike-0x host urls.
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/openstreetmap-website/pull/1117

-- Commit Summary --

  * Add Host: to robots.txt

-- File Changes --

    M public/robots.txt (2)

-- Patch Links --

https://github.com/openstreetmap/openstreetmap-website/pull/1117.patch
https://github.com/openstreetmap/openstreetmap-website/pull/1117.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/1117
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20151226/629b3b94/attachment.html>