[OSM-dev] Google doesn't find new wiki urls

[Grant Slater]

Ævar Arnfjörð Bjarmason wrote:
> The first thing that should be done is to ensure that MediaWiki's
> caching infrastructure, both internal (memcache, message cache etc)
> and external (squid) is correctly set up and all working optimally
> together. I discussed this briefly with TomH on IRC which told me to
> contact Grant regarding this, which I've done. I'd be happy to take a
> look at it to see if anything can be improved.
>
>   

emailed offlist...
The current squid server was meant to be a temporary loan... it needs to 
move somewhere better later.

> If it's correctly set up (anonymous user) requests should either
> bypass MW entirely or alternatively at least hit some cache or other
> for the stuff that's computationally expensive. Although there are
> some edge cases of course like when a template that's used 5 levels
> deep by a few hundred other templates needs to be invalidated. But
> from looking at RC it looks like the wiki has around 5-50 changes in
> the Template: namespace per day so it shouldn't be that unmanageable.
>
>   

"anonymous users" often have cookie set... causing cache misses. The 
machine runs squid WITHOUT X-Vary-Options patch! Squid server is used in 
production for other low usage clients.

When the templates change, load spikes on the mediawiki server. Load 
Avg: ~2 to ~18. On the single processor machine. It takes awhile to 
settle down.

Squid Log examples:
1231921832.295  15911 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Map_Features - NONE/- -
1231921832.303      0 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/DE:Map_Features - NONE/- -
1231921832.310      4 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/IT:Map_Features - NONE/- -
1231921832.318      3 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/FR:Map_Features - NONE/- -
1231921832.326      0 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Cz:Map_Features - NONE/- -
1231921832.367      3 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Ro:Map_Features - NONE/- -
1231921832.403      9 89.16.177.88 TCP_MISS/200 103 PURGE 
http://wiki.openstreetmap.org/wiki/Da:Map_Features - NONE/- -
1231921832.412      6 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Hu:Map_Features - NONE/- -
1231921832.452      5 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Ja:Map_Features - NONE/- -
1231921832.459      7 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Fi:Map_Features - NONE/- -
1231921832.480      4 89.16.177.88 TCP_MISS/404 110 PURGE 
http://wiki.openstreetmap.org/wiki/Ru:Map_Features - NONE/- -
...
1231923817.516  60008 89.16.177.88 TCP_HIT/000 0 PURGE 
http://wiki.openstreetmap.org/wiki/DE:Map_Features - NONE/- -

/ Grant