[OSM-dev] scaling

Frederik Ramm frederik at remote.org
Mon Jan 10 09:32:57 GMT 2011


Steve,

    another though re. your original post;

On 01/09/11 22:38, SteveC wrote:
> Could we accept the edit traffic? No, far too much. Could we provide
> a good user experience, clearly no. Could they help us scale? No they
> would be viewed as taking over on any kind of timescale they needed.
> Could they host us? Again no, it would be too slow of a process and
> it'd be a takeover and the community would probably reject it.

I think something that's entirely feasible and in no way a "takeover" 
would be setting up a number of very fast read-only mirrors that fully 
support all the API read calls - most importantly the "map" call but 
also the changeset download, full way/full relation download, and 
individual object access. (Along the lines of what Nic said, but it 
doesn't necessarily require special software.)

The technology is there - albeit a bit clumsy: consume minutely diffs 
with Osmosis and run a read-only rails port on the resulting database. 
Scaling is unlimited - set up 100 of these boxes with a load balancer in 
front, maybe add an internal HTTP proxy so you don't request the 
minutely diffs 100 times but only once - easy.

The only thing we'd have to change to make use of such an infrastructure 
is that editors would have to learn to read from one server and write to 
another. That's a very small change.

The risk of conflicts during upload would be increased but that increas 
would not come from this technology - the technology would introduce a 
minimal extra delay that is negligible compared to the average length of 
an edit session; the increase would come from more people editing at the 
same time and it would be just the same with any other method of scaling.

So, in addition to providing a lot of iron for the read access caching, 
those who want to turbo-charge OSM would perhaps have to look into 
improving conflict handling with the major editors, or implement some 
kind of conflict resolution in the API, but again that's not really a 
database scaling issue...

Bye
Frederik



More information about the dev mailing list