[openstreetmap-website] Updated robots.txt for new routes (no more /browse) (#761)

[Grant]

I don't think we need the Disallows anymore, except maybe the Disallow: /api/

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/openstreetmap-website/pull/761#issuecomment-46365184
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/rails-dev/attachments/20140617/102ece6f/attachment.html>