OSM's future. Was: Re: [Openstreetmap] OSM Performance is terrible

[SteveC]

* @ 01/02/06 10:57:57 PM openstreetmap-L at gj0.net wrote:
> <rant>
> I haven't done much on OSM for a few weeks.  The performance over the
> December holiday period got so bad that the site was unusable.  So bad
> that I avoided OSM for a whole month.  I've seen that there have been
> some recent changes so tonight I felt motivated to have another go.
> 
> In about six hours of trying I achieved almost nothing.
> 
> The speed of the OSM website is terrible.
> </rant>
> 
> WHAT CAN I DO TO HELP?

It's a fair question, and I'm sorry it hasn't been usable for you and
everyone else.

I and other people here have been quietly looking at _sustainable_ ways
of getting OSM fast and reliable hosting. Let's look at OSM right now.
It runs on 5 computers. www, tile, db, backup and dev. All of these
machines are hosted at the university I went to and worked at. Past
tense.

They are for lack of a better word, borrowed. Apart from dev
which is Nicks. The hosting is also borrowed and the machines sit in an
office being noisy and annoying people. Given that I'm not there any
more, questions are starting to arise about these machines.

Even at 5 OK-spec machines with excellent JANET hosting, it's not that
fast. So we probably need another 3-5 excellent machines. And these
machies, for the most part, need to be physically close to each other
because of the interplay of the database, tile creation and the applet.

The type of bandwidth we're using is huge. The tile server ships a lot
of data around, as does www. Commercial hosting would probably run in
to hundreds of pounds a month.

As a rough guide then, we need 10 well specced machines. Say 7 to 10
thousand pounds worth of kit and perhaps 3 to 5 thousand pounds of
hosting for the first year. We're not, yet, wikipedia and we don't have
a sugar daddy to pay for this. I'm guessing we can't just raise that
amount from the mailing list.

We're persuing multiple routes. We're trying to get a complete snapshot
of landsat out of the JPL guys[1] so we're not just their proxy. This
make distributing tiles much, much easier. I'm trying to get a formal
hosting arrangment with multiple people whereby we can use donated or
new hardware. We've also been looking at longer term arrangements in the
style that wikipedia and so on are using. I've been making funding
applications to various tech, community and art funding bodies in the
UK.

This all takes time and some politicking (dictionary.com claims that's a
real word!), hence my caution from telling you all about it. Please
don't think OSM's slowness is ignored at all, but it is starting to get
a bit silly.

An aside - The Ordnance Survey
------------------------------

As an aside... I want to communicate to you the way geo companies in the
UK generally feel about OSM. They tend to love it and want to support
it. They're terrified of the Ordnance Survey though. The OS is the sun
to which companies, education establishments and charities face. Without
the OS, they don't have any data.

I'm not an expert. There are people on this list that are. There are
people from the OS on the list, too. My experience is that people are
_extremely_ reluctant to support anything which would jeopardise their
license and relationship with the OS. I'm not sure whether this is a
real or perceived threat, but I'd tend to the former.

So I, personally, keep running in to people who say something like 'well
I would help you, but..' And that's frustrating. It was also surprising
to me just how widespread that feeling is, from PhD students to
cartographers to massive companies.

I don't like to paint the OS as a big evil monster, it's not true or
very constructive and I've always found Ed Parsons [2] to be a top
bloke. It's just the situation I find OSM, and hence myself, in.

I'll write a book about it all one day.

Back to the rant
----------------

This is all the traditional OSS way of going.

Some non-traditional things for discussion:

* Google Ads on the website
* CC-non commercial license so we can sell the maps

We have a little money from the posters. My efforts so far trying to
secure hosting have had limited real world impact so I'm ready to ask
all of you what we can do. Perhaps I should have done this in the first
place.

So, back to the question:

> WHAT CAN I DO TO HELP?

1) Hosting for 7-10 machines
2) 7-10 machines

We can get together some new and donated equipment with a funding drive.
I don't have a good idea how much we could raise? Funding would need to
run in to thousands for the higher-end machines (eg, not dev).

How much donated equipment do you think we can put together?

Hosting. Do you know a webhost or university that will offer us hosting
_formally_? They need to have more than one person who'll be responsible
for the boxes and they'll probably have to sign some ToS :-(

Some bits that can be hosted elsewhere: we can move the wiki to
wiki.openstr... and svn to svn.open... if people are able to host these,
get in touch.

If possible, please get in touch over the mailing list so others can
help. Ha, and the mailing list / MX is another thing we can host
seperately.

[1] http://onearth.jpl.nasa.gov/
[2] http://edparsons.com/

have fun,

SteveC steve at asklater.com http://www.asklater.com/steve/