[openstreetmap/openstreetmap-website] Expand robots.txt to stop pages robots are getting stuck in (PR #6875)

[Paul Norman]

@pnorman commented on this pull request.



>  Allow: /user/
-Disallow: /traces/tag/
-Disallow: /traces/page/
+Disallow: /traces

This is one of the most commonly requested pages by bots as they try to get all the pages.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/6875#pullrequestreview-3928331719
You are receiving this because you are subscribed to this thread.

Message ID: <openstreetmap/openstreetmap-website/pull/6875/review/3928331719 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20260311/fc2a6ff8/attachment.htm>