<p>Add a Host: which is an extension to the robots.txt standard. <a href="https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host">https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host</a></p>
<p>Googlebot particularly seems to be incorrectly indexing the spike-0x host urls.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117'>https://github.com/openstreetmap/openstreetmap-website/pull/1117</a></p>
<h4>Commit Summary</h4>
<ul>
<li>Add Host: to robots.txt</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/openstreetmap-website/pull/1117/files#diff-0">public/robots.txt</a>
(2)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117.patch'>https://github.com/openstreetmap/openstreetmap-website/pull/1117.patch</a></li>
<li><a href='https://github.com/openstreetmap/openstreetmap-website/pull/1117.diff'>https://github.com/openstreetmap/openstreetmap-website/pull/1117.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/openstreetmap/openstreetmap-website/pull/1117">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/ABWnLRZpsdG5IKtM9d4Vz4LnYlnXhs1Dks5pTyC5gaJpZM4G7gz8.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/openstreetmap/openstreetmap-website/pull/1117"></link>
<meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>