[openstreetmap-website] Updated robots.txt for new routes (no more /browse) (#761)

[Štefan Baebler]

Updated for changes done in
3cd5f45e08d977d04a778ab8802f71df85edc314 and
61bb31ebdddc9e7c317a1c6784a68874d5a0ef80

Test search queries listing the pages that were not suppose to be indexed by robots:
https://www.google.si/search?q=site%3Awww.openstreetmap.org+inurl%3Auser+inurl%3Ahistory
https://www.google.si/search?q=site%3Awww.openstreetmap.org+inurl%3Anode+inurl%3Ahistory
You can merge this Pull Request by running:

  git pull https://github.com/stefanb/openstreetmap-website robotstxt

Or you can view, comment on it, or merge it online at:

  https://github.com/openstreetmap/openstreetmap-website/pull/761

-- Commit Summary --

  * Updated robots.txt for new routes (no more /browse)

-- File Changes --

    M public/robots.txt (14)

-- Patch Links --

https://github.com/openstreetmap/openstreetmap-website/pull/761.patch
https://github.com/openstreetmap/openstreetmap-website/pull/761.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/761
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20140616/a86173eb/attachment.html>