[OSM-dev] Google doesn't find new wiki urls

[Grant Slater]

Erik Johansson wrote:
> Firefishy unblocked google for now. I'm guessing it's too much trouble
> to be worth it to put the wiki config+robots.txt in SVN. (I did look
> for it before asking)
>
>   

I had unblocked googlebot, the load spiked and wiki became unresponsive 
again in the evening peak. I have now restricted access to non-peak 
hours, which seems to work ok.

Robots.txt here:
http://wiki.openstreetmap.org/robots.txt
But access control is managed by Squid.

Mediawiki config fairly standard:
    $wgUseMemCached = true;
    $wgMainCacheType = CACHE_MEMCACHED;
    $wgMessageCacheType = CACHE_MEMCACHED;
    $wgParserCacheType = CACHE_MEMCACHED;
    $wgSessionsInMemcached = true; # optional
    $wgHitcounterUpdateFreq = 100;
    $wgShowIPinHeader       = false;
    $wgUseSquid = true;
    $wgSquidServers = array('89.145.68.232');

Any other questions?

/ Grant