[openstreetmap/openstreetmap-website] Add noindex meta tag to diary index pages (#3029)

[Tobias]

I always thought robots.txt an noindex-metatags would just be the same but different methods. However, I find the meta tag easier to understand and validate its correctness. However, reading on the google help page, it sounds like the robots.txt way is actually seen as something different by google and they recommend using the meta-tag (or http-verbs).

So IMO this is the way to go …

https://developers.google.com/search/docs/advanced/robots/intro?hl=en
> You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. **If you want to block your page from search results, use another method such as password protection or a noindex directive.**
> https://developers.google.com/search/reference/robots_meta_tag

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/3029#issuecomment-757330508
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20210109/a30bc735/attachment.htm>