[OSM-dev] scaling

[Kai Krueger]

Frederik Ramm wrote:
> 
> This was not purely a techical issue. If we were set up, technically, to 
> handle something like what you're describing here, the "eternal 
> september" effect would kill off the community for good.
> 

In the case of something as extreme as "turns all our 
yearly statistics into daily or weekly numbers", yes that might be a
problem.

But currently still about 30 - 50% of all users are in Germany (
http://neis-one.org/wp-content/uploads/2010/06/OSM_User_Planet_20100610.png 
are there any more recent statistics?). If all the countries in OSM edited
at the same pace as Germany, we would fairly surely have a scalability
problem with our current hardware / software. The German growth hasn't
caused a massive amount of social scalability issues. So why do you think
scaling OSM to by an order of magnitude (which would easily be the case if
more countries got to the German level) would be troublesome?

So unless you don't want to expand at all, thinking about scalability is an
important issue and probably shouldn't be left till all of the servers are
creaking and failing under the load.

And it is not that we don't ever see any overload problems with our current
setup even with current editing levels, or that everything is so massively
over provisioned that one doesn't observe any slowdown during Sunday
afternoon peak times.  



Frederik Ramm wrote:
> 
> So in a way, we must be glad that we have this technical limitation,
> 

Making the API so slow that people don't want to edit anymore is not the
right solution to the social issue, or to any other issue either.



Frederik Ramm wrote:
> 
>> Even if people weren't privately proposing notching up our traffic a
>> few orders of magnitude
> 
> Please stop thinking of this as a technical issue. It isn't!
> 

Well, it is also a technical issue. And if you haven't solved the technical
issue, there is also limited point in solving the social one. (Of cause that
is true the other way round too, although the technical issue imho arises
much sooner than the social one)

> But the direction of supporting and encouraging basic
> things like scaling is I think well within the bounds.
> 
> I haven't a clue what we should use to scale horizontally.


Frederik Ramm wrote:
> 
> I think "encouraging scaling" is within the bounds of what OSMF could 
> think about. Whether or not such scaling needs to be horizontal, 
> vertical, deep-bore or whatever other method is currently en vogue, 
> that's perhaps something best left to the TWG.
> 
It will obviously need strong support and input from the TWG, but a solution
doesn't necessarily need to be initiated by or come from the TWG, but can
come from the wider technical community in OSM.


Frederik Ramm wrote:
> 
>> So, how do we get from here to there? Speaking strictly personally, I
>> think one of the best uses of funds in or out of OSM has been bug
>> bounties. 
> 
> Does anyone have an example where this has worked in OSM? I always 
> perceived these bounties as your personal hobby horse that never worked 
> for us but maybe my thinking has clouded my vision here.
> 
> Personally, I think that people tend to be encouraged by the hope that 
> their solution might become THE solution used in OSM. Tell a hacker that 
> he gets $1000 if he gets something to work, and he'll say "oh well, 
> let's leave that to those who need the cash". Tell a hacker that you'll 
> roll out his idea big time if he proves it can work, and he'll happily 
> do it...
> 
Yes, I would agree with you there. Non-monetary incentives and
encouragements like the ones you mentioned are probably more important and
if done right could imho get a lot more of development going.


Frederik Ramm wrote:
> 
>> So, what do you think? And if you agree it's worth doing, how do we
>> achieve it either as individuals or the board or companies supporting
>> it?
> 
The first thing would probably be to ensure that there is a universal will
to achieve the goal amongst those involved and that any solutions proposed
or implemented don't go the way of the front page redesign, the logo
redesign or some of the other not quite so successful proposed changes in
the past.
-- 
View this message in context: http://gis.638310.n2.nabble.com/scaling-tp5905305p5905537.html
Sent from the Developer Discussion mailing list archive at Nabble.com.