[OSM-dev] scaling

[Matt Amos]

On Mon, Jan 10, 2011 at 1:31 PM, Andy Allan <gravitystorm at gmail.com> wrote:
> On Mon, Jan 10, 2011 at 12:44 PM, Robert Scott <lists at humanleg.org.uk> wrote:
>> On Monday 10 January 2011, Kai Krueger wrote:
>>> Depending on how far you really want to scale, I think a lot of the
>>> necessary components are already in place. If none of the "out of the box"
>>> solutions such as the new Postgresql 9.0 replication mechanism  work, we
>>> would possibly get a fair distance by splitting out reads and writes onto
>>> separate db servers.
>>
>> This is how the postgres 9 replication works. The replicating servers become "hot standbys" which you can use for read requests. So in theory the read requests could be scaled quite easily once set up. Atomicity of the API would potentially suffer though.
>
> We need to be careful about what purposes we scale read requests. A
> lot of the read-load on the API is using the data for non-editing
> purposes (such as rendering or other 'cool' things that need /map
> calls) and this should be avoided[1]. The advantage of making sure
> that read-scaling for non-editing purposes is *not* via db-replication
> is that anyone (and everyone) can add their own servers to scale
> things out. For example, everything that currently fetches live data
> via the diffs at planet.openstreetmap.org (or a service that depends
> on them, like xapi, geofabrik extracts etc) is a Good Thing, and
> everything that currently fetches live read-only data via
> osm.org/api/0.6/map is a Bad Thing.
>
> It's a bad thing since there's only so much hardware the OSMF can
> own/host/run, but there's nothing stopping 50 other organisations
> running cool stuff based off of planet.openstreetmap.org

it seems postgres 9 isn't able to do temporary table creation on
hot-standby servers [1]. maybe this is something that'll be fixed in
future versions (but it doesn't seem to be on the TODO [2]) or maybe
it's an opportunity to put a bounty to good use.

cheers,

matt

[1] http://www.postgresql.org/docs/9.0/interactive/hot-standby.html#HOT-STANDBY-USERS
[2] http://wiki.postgresql.org/wiki/Todo