[OSM-dev] Google doesn't find new wiki urls
Tom Hughes
tom at compton.nu
Wed Jan 14 12:36:58 GMT 2009
Ævar Arnfjörð Bjarmason wrote:
> On Wed, Jan 14, 2009 at 10:36 AM, Tom Hughes <tom at compton.nu> wrote:
>
>> The googlebot is probably blocked as bots tend to kill the wiki due to
>> the extreme crapness of mediawiki.
>
> Bots happily hit Wikipedia all day without killing it, is the OSM wiki
> set up equivalently? I.e. with a cache in front of it, memcache to
> cache various things within MW etc?
Yes, we're setup a little differently.
We have one little Atom based machine. They have a metric buttload of
database servers serving pages to ten metric buttloads of web servers
fronted by about a thousand metric buttloads of squid caches.
Basically wikipedia have "solved" the problem of mediawiki being a crock
by throwing ludicrous amounts of hardware at it.
It doesn't help that we have some ridiculously complicated pages like
map features which are not only hugely complicated with masses of
templates but also get changed very frequently.
Tom
--
Tom Hughes (tom at compton.nu)
http://www.compton.nu/
More information about the dev
mailing list