[OSM-dev] Google doesn't find new wiki urls

[Grant Slater]

Maarten Deen wrote:
> Grant Slater wrote:
>   
>> Erik Johansson wrote:
>>     
>>> Firefishy unblocked google for now. I'm guessing it's too much trouble
>>> to be worth it to put the wiki config+robots.txt in SVN. (I did look
>>> for it before asking)
>>>       
>> I had unblocked googlebot, the load spiked and wiki became unresponsive
>> again in the evening peak. I have now restricted access to non-peak
>> hours, which seems to work ok.
>>     
>
> Have you tried the Google webmaster tools? You can specify the rate of crawl
> there. And there are a lot of statistics there that may be useful.
>   

Yup. I tried tweaking it awhile back. Still had problems.

Sitemap did help.
http://wiki.openstreetmap.org/sitemap-index-wiki.xml

The server is only an Intel Atom 230. It runs Wiki, SVN, All outbound 
email and Lists.

/ Grant