<p>Add a Host: which is an extension to the robots.txt standard. <a href="https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host">https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host</a></p>

<p>Googlebot particularly seems to be incorrectly indexing the spike-0x host urls.</p>

<hr>

<h4>You can view, comment on, or merge this pull request online at:</h4>
<p>  <a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117'>https://github.com/openstreetmap/openstreetmap-website/pull/1117</a></p>

<h4>Commit Summary</h4>
<ul>
  <li>Add Host: to robots.txt</li>
</ul>

<h4>File Changes</h4>
<ul>
  <li>
    <strong>M</strong>
    <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1117/files#diff-0">public/robots.txt</a>
    (2)
  </li>
</ul>

<h4>Patch Links:</h4>
<ul>
  <li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117.patch'>https://github.com/openstreetmap/openstreetmap-website/pull/1117.patch</a></li>
  <li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117.diff'>https://github.com/openstreetmap/openstreetmap-website/pull/1117.diff</a></li>
</ul>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1117">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/ABWnLRZpsdG5IKtM9d4Vz4LnYlnXhs1Dks5pTyC5gaJpZM4G7gz8.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
  <link itemprop="url" href="https://github.com/openstreetmap/openstreetmap-website/pull/1117"></link>
  <meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>