[OSM-dev] scaling

[Nic Roets]

On Mon, Jan 10, 2011 at 3:43 AM, SteveC <steve at asklater.com> wrote:
> What you're effectively saying is move certain things to something that looks like xapi?

Yes.

Note that the rate of edits to the road network is limited to the
amount of real world changes to the road network plus a once off catch
up amount. Even if OSM becomes popular in the global South (India,
China, etc) (something that not even Wikipedia achieved), then the
increase will be closer to 4 fold, rather than 100 fold. If everyone
in the world add their home address as a node, it will barely double
the planet.

Edits to buildings can cause a problem. But I don't see buildings as
critical to the success of OSM, so we can take steps to deemphasize
them if we have to.

And the evidence is that the number of changesets have been growing
linearly at +- 300,000 per month since late 2009.

Read requests however will explode and keep growing exponentially for
regions where OSM is becoming the authoritative source for
geographical data.

--

I think Frederik is right. If we grow 100 fold or even 10 fold,
trolls, vandals and newbies will be our biggest problem.

> Fred
>
> May I paraphrase? You push here, and say in other replies, that you're concerned about the social impact (of a much larger community).
>
> As the guy who implemented the first friend features and was called an idiot for it, and doing a lot of community building, I think I know a thing or two about that stuff.
>
> I read the other day that proof-by-contradiction is too easy and you should try for others, but it's too easy to pass up. Let's say you are so concerned about the social aspects of OSM. Isn't it a contradiction given that there is no community on talk@ any more, and that the community is basically (give or take) the same for the in-person meetings it's been for what, 3 years? And we're clearly moving to tools that can handle radically bigger communities more efficiently like the osm help site?
>
> I don't think you (or anyone else) is much up for fixing the community on talk@ despite the obvious things to do, I don't think growing radically the community will impact the other things outlined above, therefore I don't think the community argument flies.
>
> Or am I wrong?
>
>
>
> On Jan 9, 2011, at 2:35 PM, Frederik Ramm wrote:
>
>> Steve,
>>
>> SteveC wrote:
>>> I don't mean just another big import. Sadly I can't be public
>>> but I had a conversation with a large company over a year ago (no,
>>> it's not MS or CM) who  speculated about putting OSM on the front
>>> page of their maps product, which would approximately turn all of our
>>> yearly statistics to daily or weekly numbers. We went through a
>>> decision tree about how that could happen. Every leaf node on that
>>> tree came back as basically we couldn't do it.
>>
>> This was not purely a techical issue. If we were set up, technically, to handle something like what you're describing here, the "eternal september" effect would kill off the community for good.
>>
>> So in a way, we must be glad that we have this technical limitation, because it protects us against someone dumping 1000s of new users every week onto our mailing lists, IRC channels, forums, onto help.osm.org, the Wiki, and int our regular pub meets.
>>
>>> If you think through it,
>>> within any reasonable time frame (like 6-12 months) it's very hard to
>>> make that happen, and so you may as well go build your own things.
>>> Which I think sucks and is a loss for OSM.
>>
>> I honestly think that the project would degrade tremendously were something like this to happen. We can handle growth, and I also believe we can handle super-linear growth, but something that "turns all our yearly statistics into daily or weekly numbers" would be the end of everything that is sane about OSM.
>>
>>> Now this conversation has come up a few more times recently with
>>> other large mapping companies. And I feel like I'm rehashing those
>>> conversations above. I'd love to be public about it, but those
>>> companies aren't ready to talk yet.
>>
>> Good for them because they would have to take the heat for killing OSM - or nonchalantly thinking about doing that.
>>
>>> Even if people weren't privately proposing notching up our traffic a
>>> few orders of magnitude
>>
>> Please stop thinking of this as a technical issue. It isn't!
>>
>>> But the direction of supporting and encouraging basic
>>> things like scaling is I think well within the bounds.
>>> I haven't a clue what we should use to scale horizontally.
>>
>> I think "encouraging scaling" is within the bounds of what OSMF could think about. Whether or not such scaling needs to be horizontal, vertical, deep-bore or whatever other method is currently en vogue, that's perhaps something best left to the TWG.
>>
>> If OSMF board wants to realistically construct a scenario that has the number of new users, and the amount of edits and traffic, multiplied by 100 in the frame of a few months, then I'm sure that TWG can provide ideas how to get there. But you would *have* to find someone who creates that scenario for the social side and looks into the issues I've brought up above or else we'll be fucked on a level that's much harder to fix than a slow database.
>>
>> I don't know how much your personal view might be tainted by the US perspective. Compared to Europe, ans especially taking into account the population base, community size in the US is still negligible. You might be tempted to think that you can play va banque there as there isn't a lot to lose. If that's the case then maybe we should work on providing more separation - allowing you to supercahrge and horizontally scale OSM-US with the help of one megacorp or the other, but keeping us here in Europe out of harm's way.
>>
>> (Indeed one way of scaling would be to "devolve" the central database - have several of them for different regions. That would make it more difficult for people to write bots that wreak havoc with the world-wide dataset but I wouldn't consider that a loss ;)
>>
>>> So, how do we get from here to there? Speaking strictly personally, I
>>> think one of the best uses of funds in or out of OSM has been bug
>>> bounties.
>>
>> Does anyone have an example where this has worked in OSM? I always perceived these bounties as your personal hobby horse that never worked for us but maybe my thinking has clouded my vision here.
>>
>> Personally, I think that people tend to be encouraged by the hope that their solution might become THE solution used in OSM. Tell a hacker that he gets $1000 if he gets something to work, and he'll say "oh well, let's leave that to those who need the cash". Tell a hacker that you'll roll out his idea big time if he proves it can work, and he'll happily do it...
>>
>>> So, what do you think? And if you agree it's worth doing, how do we
>>> achieve it either as individuals or the board or companies supporting
>>> it?
>>
>> First of all, stop talking of the kind of explosive growth you're talking about. Set yourselves a realistic growth scenario - I'd say something like "number of contributors & data size double every year" or so. Sit down with TWG and find out how long the traditional approach will hold out, and how and when to scale, then make plans for that. I'm sure TWG will be mature enough to say what they can do and what they can't, and where they will need input & concepts from the project at large. Find suitable ways to solicit such.
>>
>> If you are looking for a project that can support the kind of explosive growth you envisage - factor 100 in a few months -, then start a new project, one that relies much less on individual human beings. A project where you can simply scale by adding another server, without also having to tear down the pub and erect a town hall in its place.
>>
>> Bye
>> Frederik
>>
>> --
>> Frederik Ramm  ##  eMail frederik at remote.org  ##  N49°00'09" E008°23'33"
>>
>
> Steve
>
> stevecoast.com
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>