[OSM-talk] [OSM-dev] donating read-only api-mirrors

Matt Amos zerebubuth at gmail.com
Fri Feb 6 18:25:36 GMT 2009


On Fri, Feb 6, 2009 at 5:20 PM, Marcus Wolschon <Marcus at wolschon.biz> wrote:
> On Fri, Feb 6, 2009 at 4:35 PM, Tom Hughes <tom at compton.nu> wrote:
>> Erik Johansson wrote:
>>
>>> How much of the DB load comes from the read only part of the  API, and
>>> what if you remove the area limit on the map call?
>>
>> If I remove the area limit then somebody will do a massive query that
>> will suck up all the memory on the machine and everything will die.
>>
>> That limit is not (primarily) about the cost of gathering the required
>> data, though obviously that might become an issue as well, it's about
>> the fact that we are holding the whole result set in memory (several
>> times over in fact) on the rails server and those servers only have a
>> finite amount of memory.
>
> I'm not sure I understand why you do this.
> The most expensive call would probably be /map with a bounding-box.
> You query all the node, stream them out and keep the nodeIDs
> (but not the whole resultset with lat, lon, version, ...) in memory.
> Query the ways that use these nodes, keeping their IDs in memory.
> Then Query the relations, stream them out and forget about the nodes
> and ways while doing that.

in an ideal world. however, there are a few issues; Frederik has
already raised the point about sorting by ID, Rails doesn't (yet?)
support streaming results and you cannot indicate an error condition
(e.g: MySQL dies, bad UTF-8) in the response header once you've
already sent part of the body.

> Anyway,  the question was on how much of the  current load is done by
> non-writing queries (not just /map). Do we have an answer for that?

not directly, but by comparing

http://munin.openstreetmap.org/openstreetmap/db.openstreetmap-load.html
http://munin.openstreetmap.org/openstreetmap/db.openstreetmap-mysql_queries.html

we can see that about 75% of all queries are selects, but that times
of high load appear better correlated with inserts and deletes. it is
difficult to quantify how much load is purely due to reads, as part of
the load may be due to contention between the reads and writes.

cheers,

matt




More information about the talk mailing list