[OSM-dev] Google doesn't find new wiki urls

Tom Hughes tom at compton.nu
Wed Jan 14 12:36:58 GMT 2009


Ævar Arnfjörð Bjarmason wrote:

> On Wed, Jan 14, 2009 at 10:36 AM, Tom Hughes <tom at compton.nu> wrote:
>
>> The googlebot is probably blocked as bots tend to kill the wiki due to
>> the extreme crapness of mediawiki.
> 
> Bots happily hit Wikipedia all day without killing it, is the OSM wiki
> set up equivalently? I.e. with a cache in front of it, memcache to
> cache various things within MW etc?

Yes, we're setup a little differently.

We have one little Atom based machine. They have a metric buttload of 
database servers serving pages to ten metric buttloads of web servers 
fronted by about a thousand metric buttloads of squid caches.

Basically wikipedia have "solved" the problem of mediawiki being a crock 
by throwing ludicrous amounts of hardware at it.

It doesn't help that we have some ridiculously complicated pages like 
map features which are not only hugely complicated with masses of 
templates but also get changed very frequently.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/




More information about the dev mailing list