[OSM-dev] scaling

[Frederik Ramm]

Steve,

SteveC wrote:
> I don't mean just another big import. Sadly I can't be public
> but I had a conversation with a large company over a year ago (no,
> it's not MS or CM) who  speculated about putting OSM on the front
> page of their maps product, which would approximately turn all of our
> yearly statistics to daily or weekly numbers. We went through a
> decision tree about how that could happen. Every leaf node on that
> tree came back as basically we couldn't do it.

This was not purely a techical issue. If we were set up, technically, to 
handle something like what you're describing here, the "eternal 
september" effect would kill off the community for good.

So in a way, we must be glad that we have this technical limitation, 
because it protects us against someone dumping 1000s of new users every 
week onto our mailing lists, IRC channels, forums, onto help.osm.org, 
the Wiki, and int our regular pub meets.

> If you think through it,
> within any reasonable time frame (like 6-12 months) it's very hard to
> make that happen, and so you may as well go build your own things.
> Which I think sucks and is a loss for OSM.

I honestly think that the project would degrade tremendously were 
something like this to happen. We can handle growth, and I also believe 
we can handle super-linear growth, but something that "turns all our 
yearly statistics into daily or weekly numbers" would be the end of 
everything that is sane about OSM.

> Now this conversation has come up a few more times recently with
> other large mapping companies. And I feel like I'm rehashing those
> conversations above. I'd love to be public about it, but those
> companies aren't ready to talk yet.

Good for them because they would have to take the heat for killing OSM - 
or nonchalantly thinking about doing that.

> Even if people weren't privately proposing notching up our traffic a
> few orders of magnitude

Please stop thinking of this as a technical issue. It isn't!

> But the direction of supporting and encouraging basic
> things like scaling is I think well within the bounds.
> 
> I haven't a clue what we should use to scale horizontally.

I think "encouraging scaling" is within the bounds of what OSMF could 
think about. Whether or not such scaling needs to be horizontal, 
vertical, deep-bore or whatever other method is currently en vogue, 
that's perhaps something best left to the TWG.

If OSMF board wants to realistically construct a scenario that has the 
number of new users, and the amount of edits and traffic, multiplied by 
100 in the frame of a few months, then I'm sure that TWG can provide 
ideas how to get there. But you would *have* to find someone who creates 
that scenario for the social side and looks into the issues I've brought 
up above or else we'll be fucked on a level that's much harder to fix 
than a slow database.

I don't know how much your personal view might be tainted by the US 
perspective. Compared to Europe, ans especially taking into account the 
population base, community size in the US is still negligible. You might 
be tempted to think that you can play va banque there as there isn't a 
lot to lose. If that's the case then maybe we should work on providing 
more separation - allowing you to supercahrge and horizontally scale 
OSM-US with the help of one megacorp or the other, but keeping us here 
in Europe out of harm's way.

(Indeed one way of scaling would be to "devolve" the central database - 
have several of them for different regions. That would make it more 
difficult for people to write bots that wreak havoc with the world-wide 
dataset but I wouldn't consider that a loss ;)

> So, how do we get from here to there? Speaking strictly personally, I
> think one of the best uses of funds in or out of OSM has been bug
> bounties. 

Does anyone have an example where this has worked in OSM? I always 
perceived these bounties as your personal hobby horse that never worked 
for us but maybe my thinking has clouded my vision here.

Personally, I think that people tend to be encouraged by the hope that 
their solution might become THE solution used in OSM. Tell a hacker that 
he gets $1000 if he gets something to work, and he'll say "oh well, 
let's leave that to those who need the cash". Tell a hacker that you'll 
roll out his idea big time if he proves it can work, and he'll happily 
do it...

> So, what do you think? And if you agree it's worth doing, how do we
> achieve it either as individuals or the board or companies supporting
> it?

First of all, stop talking of the kind of explosive growth you're 
talking about. Set yourselves a realistic growth scenario - I'd say 
something like "number of contributors & data size double every year" or 
so. Sit down with TWG and find out how long the traditional approach 
will hold out, and how and when to scale, then make plans for that. I'm 
sure TWG will be mature enough to say what they can do and what they 
can't, and where they will need input & concepts from the project at 
large. Find suitable ways to solicit such.

If you are looking for a project that can support the kind of explosive 
growth you envisage - factor 100 in a few months -, then start a new 
project, one that relies much less on individual human beings. A project 
where you can simply scale by adding another server, without also having 
to tear down the pub and erect a town hall in its place.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"