[OSM-dev] scaling

[Nic Roets]

I'm not the one funding the hardware nor handing out the bounties,
BUT I would like to see a little bit more accounting / budgeting, as in :
1. The current hardware costs $X and can serve Y million map queries per year,
2. A XAPI server costs ... and can serve ... million map queries per year,
3. We estimate that for every NNN map queries we serve to example.com,
we get a user to make one correction to the DB. This compares to MMM
map queries per edit for visitors to osm.org

Part of the reason for 1 and 2 is that there is very little
competition in the large server market and that translate into poor
value for money. So economies of scale may not kick in.

Then we can estimate how much amount of money will be saved by a rewrite in C.

Then we can say to corporates and webmasters that they will be a drain
on the project unless they donate $D to us.

No need to be accurate to the second decimal. Or the first decimal.

Regards,
Nic

On Mon, Jan 10, 2011 at 3:31 PM, Andy Allan <gravitystorm at gmail.com> wrote:
> On Mon, Jan 10, 2011 at 12:44 PM, Robert Scott <lists at humanleg.org.uk> wrote:
>> On Monday 10 January 2011, Kai Krueger wrote:
>>> Depending on how far you really want to scale, I think a lot of the
>>> necessary components are already in place. If none of the "out of the box"
>>> solutions such as the new Postgresql 9.0 replication mechanism  work, we
>>> would possibly get a fair distance by splitting out reads and writes onto
>>> separate db servers.
>>
>> This is how the postgres 9 replication works. The replicating servers become "hot standbys" which you can use for read requests. So in theory the read requests could be scaled quite easily once set up. Atomicity of the API would potentially suffer though.
>
> We need to be careful about what purposes we scale read requests. A
> lot of the read-load on the API is using the data for non-editing
> purposes (such as rendering or other 'cool' things that need /map
> calls) and this should be avoided[1]. The advantage of making sure
> that read-scaling for non-editing purposes is *not* via db-replication
> is that anyone (and everyone) can add their own servers to scale
> things out. For example, everything that currently fetches live data
> via the diffs at planet.openstreetmap.org (or a service that depends
> on them, like xapi, geofabrik extracts etc) is a Good Thing, and
> everything that currently fetches live read-only data via
> osm.org/api/0.6/map is a Bad Thing.
>
> It's a bad thing since there's only so much hardware the OSMF can
> own/host/run, but there's nothing stopping 50 other organisations
> running cool stuff based off of planet.openstreetmap.org
>
> Cheers,
> Andy
>
> [1] It's against the policy at
> http://wiki.openstreetmap.org/wiki/API_usage_policy , for a start.
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>