[OSM-dev] scaling

[Andy Allan]

On Mon, Jan 10, 2011 at 12:44 PM, Robert Scott <lists at humanleg.org.uk> wrote:
> On Monday 10 January 2011, Kai Krueger wrote:
>> Depending on how far you really want to scale, I think a lot of the
>> necessary components are already in place. If none of the "out of the box"
>> solutions such as the new Postgresql 9.0 replication mechanism  work, we
>> would possibly get a fair distance by splitting out reads and writes onto
>> separate db servers.
>
> This is how the postgres 9 replication works. The replicating servers become "hot standbys" which you can use for read requests. So in theory the read requests could be scaled quite easily once set up. Atomicity of the API would potentially suffer though.

We need to be careful about what purposes we scale read requests. A
lot of the read-load on the API is using the data for non-editing
purposes (such as rendering or other 'cool' things that need /map
calls) and this should be avoided[1]. The advantage of making sure
that read-scaling for non-editing purposes is *not* via db-replication
is that anyone (and everyone) can add their own servers to scale
things out. For example, everything that currently fetches live data
via the diffs at planet.openstreetmap.org (or a service that depends
on them, like xapi, geofabrik extracts etc) is a Good Thing, and
everything that currently fetches live read-only data via
osm.org/api/0.6/map is a Bad Thing.

It's a bad thing since there's only so much hardware the OSMF can
own/host/run, but there's nothing stopping 50 other organisations
running cool stuff based off of planet.openstreetmap.org

Cheers,
Andy

[1] It's against the policy at
http://wiki.openstreetmap.org/wiki/API_usage_policy , for a start.