[OSM-dev] scaling

[Serge Wroclawski]

As often with such topics, the movement of the discussion is so fast
and varied that it's hard to get a solid hold on any of it, but here's
my .02, for what it's worth. And I apologize in advance for the length
of this email, since I'm trying to respond to both the whole threat,
and specific comments made.

1. Yes, there's a problem

I think we're all in general agreement that the status quo won't work
for a whole lot longer. I've been saying this with increasing volume
since I joined the project. We're growing exponentially and I fear
that the exponential wall is going to smack us in the face sooner than
we'd like.

Having a problem isn't a terrible thing though, so let's not let
emotions run wild.

2. Identifying broad, sweeping problems without a proposed solution is silliness

If we're interested in doing something other than trolling, we have to
identify specific needs and problems. I think the thread is finally
doing that, hence jumping in now.

The main problems I see being expressed are:

a) The Read API is slow

b) The Write API is slow

c) Machines won't grow to fit the problem

d) (and I haven't seen this expressed clearly but I've only skimmed
some of the emails) OSM is not geographically dispersed.

I think these are all currently addressable problems

3. The future is going to be awesome, but it's not here yet.

I think that in the far flung future, there will absolutely be need
and utility for some kind of truly distributed mapping application
that uses lots of sources.

Unfortunately we have plenty of trouble running the existing OSM and
trying to reconcile what we have, so at this moment, a distributed map
of this type would be all noise and be of little value. The problems
with this distributed approach isn't technical, it's logical, which is
far harder to solve, so let's stick to the unified dataset model for
the time being and the problems ahead.

4. The OSMF/ Working Groups Aren't Wizards on a Hill

There seems to be a communal split in the discussion between those who
think they should take up the effort and fork, and those who think the
hard decisions belong to those in charge (OSMF/TWG, etc.).

I reject both notions. My view on OSM forking is well known, so let's
address the second point.

The Technical Working Group are very smart people, but it's up to the
community to come up with clever solutions to problems as well. Like
other software projects, sometimes you have to take the raw material,
fudge with it, and then come up with something to show.

Last year at SoTM, I did a lightening talk on using MongoDB to store
OSM data. I've largely been lax in working on the project, but I was
addressing some of the concerns I hear being expressed in this threat.
 I'd love to see others working on solutions of their own, to present
to the technical working group, to show them potential innovative
solutions, and to be part of the way forward.

5. These are good problems to have

Horizontal scaling is an interesting problem, and I think that if
there were a clever coder out there who created a good solution to
this that worked reliably, they could add some other features in the
stack.


Now let's talk about things we can do today to fix the current situation:

6. API reads can and should be made on dedicated hardware.

The favored XAPI implementation is changing, and I'd bet Ian would be
able (and willing, given the hardware) to add single object and
history support.

That's give it the ability to serve most of the read API needs.
Furthermore, with a fully baked solution, it'd be easier to
geographically dispurse the reads across the world when the time for
that comes.

I'm willing to bet that Ian's Java XAPI is faster at handling a bbox
request than Rails is, and so as soon as we can move the requests to
this implementation, the better, even if the hardware stayed the same.

7. Query optimisation

We tend to think of our queries as being fairly straightforward, but
maybe someone smart who has time could examine the logs to see if
that's the case and if there's not way that a query could be
optimized.

I know Ian worked on query optimization in the XAPI implementation.
Query optimization is difficult and relatively specialized.
Nonetheless we may be able to get even more performance, and thus
faster response times for the users, and lower load on the hardware.

8. Encoding, alternate client protocols, etc.

We do support Gzip encoded requests and I think both Josm and Potlatch
support them by default (I know Josm does for sure). Simply by
supporting gzip, I've seen a 30% speedup. Any editor which doesn't use
gzip encoding should be fixed to do so.

We may want to look at other client-server optimisations to make in
order to shorten the time that the request is delivered to the client
as well.

9. Data formats

We all love XML but I know for myself that XML is expensive to encode
to and decode properly and even a SAX parser has significant overhead.
If you think about a client needing to consume large data resources,
I'm wondering if there aren't more efficient data formats we could be
using, including something as simple as JSON, which might result in,
again, faster updates to the browser.

Longer term:

10. Geographic/Network optimised mirrors.

Once an update mechanism is set in stone, the OSMF could federate
Read-Only mirrors geographically.

11. I think we can do distributed writes.

I think we can do distributed writes.

I also know distributed writes are unpopular because of the issues
they could create, but I think if we make three changes, we could do
it.

a) Replace serial object IDs with UUIDs

Using serial interger IDs means that servers need to keep a lot of
state between them. If node IDs are effectively random, this problem
goes away.

b) Replace the changeset metaphor with a transaction metaphor

Everyone who comes to the project wants to think that a changeset is
atomic. It's confusing. I think a future OSM protocol should replace
this with a new metaphor which would be atomic, and wouldn't be able
to span multiple requests. One request would be a single transaction,
and we'd ensure that these transactions would be atomic, even if
they're slow. But even this may be largely mitigated by my third
suggestion.

c) Provide an update mechanism for the editors

While offline editing is used by folks in HOT (usually to add data!),
the vast majority of editors are editing while online. So to reduce
the chances of conflict, it would be best if the clients "checked in"
to an area, and were then sent updates of that area.

How might this work in practice? I open my editor and select an area,
and then the area I'm looking at in my screen is represented by a bbox
which I sent to the server in a long-poll request. If data is updated
in that area, the server sends me the changes and I make a new
request.

By doing this, my client has a chance to catch conflicts before
they're submitted to the server. If there is a conflict in my
submitted chances, I'm able to address it locally.

This doesn't absolutely ensure that there are no conflicts, but it
could reduce the chances.

Right now this might be too many requests for us to handle, but once
we have more servers in place, I think this would be entirely doable,
and feel natural and collaborative in the way that Google Docs or
Etherpad does.


I apologise for a long mail, and I may be just adding to the trolls,
but I think that with modest changes to our systems, we can take
ourselves pretty far, and with some moderate changes, be in a very
good position to tackle these scaling issues.

- Serge