[openstreetmap-website] Updated robots.txt for new routes (no more /browse) (#761)

[Štefan Baebler]

It is nothing new, just fixed after it got broken in the mentioned revisions.

Indexing history might not be desirable because:
* it demotes SEO-importance of current (most recent) data 
* loads the servers with irrelevant, heavy requests, leading to
* greater delays with indexing current data
* spam/vandalism remains visible even after being removed
* possible copyright violations remain visible (although this is not a way to address such issues)

The only pro-history-indexing argument would be that vandalism (eg deleting/changing name tags) would be possible to discover this way.

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/761#issuecomment-46269080
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20140616/b954b94a/attachment.html>