[openstreetmap/openstreetmap-website] Add noindex meta tag to diary index pages (#3029)

[Matija Nalis]

@tomhughes  Basically, you want search engines to index the detailed page (like for example [this page](https://www.openstreetmap.org/user/Hans%20Thompson/diary/396516)) and not index page (like for example [this page](https://www.openstreetmap.org/user/Hans%20Thompson/diary)).  The `noindex` implemented in this PR is a hint to search engine that "_this is a duplicate page that should not be indexed, as real page with same or better data is available elsewhere_".

Main reason why would you want that behavior is that index page will often differ between the time they where indexed by search engine, and the time users visit them (eg. user will publish new diary entries which would push older entries to second subpage). 
Net result of that fact is that **users searching for some term on search engine will often land on index page that no longer contains the search term they searched for**, which is annoying to users (and we presumably want to make users happy). Hinting to search engines that they should not index such index pages solves the problem, as only detailed pages get cached then.

There are also other reasons why detail ("leaf") page (instead of index page) should always be preferred:
- index page might contain shortened text instead of full text
- detailed page might get downranked if index page is also indexed by search engine
- having users pick which of the several very similar versions of page they want to visit is wasting their time
- having same content appear on multiple pages might make the whole site look more spammy (especially when some users do put spam content in diaries!)
- it leads to less load on web servers and more up-to-date data in search engines (as with hint they only index portion of the pages, they can do faster, and do not need to reindex unchanged detail pages)
- etc.

More background is explained in linked https://github.com/openstreetmap/openstreetmap-website/issues/2851#issuecomment-761879964 and rest of that ticket, and link to several [SEO](https://en.wikipedia.org/wiki/Search_engine_optimization) pages which explain the issue in more detail.

So helping search engines decide what pages are important (and which are not) helps not only search engines, but also users trying to find OSM content, as well as OSM infrastructure load / bandwidth.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/3029#issuecomment-829497586
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20210429/1b800063/attachment.htm>