[OSM-dev] Google doesn't find new wiki urls

[Maarten Deen]

Grant Slater wrote:
> Erik Johansson wrote:
>> Firefishy unblocked google for now. I'm guessing it's too much trouble
>> to be worth it to put the wiki config+robots.txt in SVN. (I did look
>> for it before asking)
>
> I had unblocked googlebot, the load spiked and wiki became unresponsive
> again in the evening peak. I have now restricted access to non-peak
> hours, which seems to work ok.

Have you tried the Google webmaster tools? You can specify the rate of crawl
there. And there are a lot of statistics there that may be useful.

Regards,
Maarten