[OSM-dev] [OSM-talk] donating read-only api-mirrors

[Marcus Wolschon]

On Fri, Feb 6, 2009 at 4:35 PM, Tom Hughes <tom at compton.nu> wrote:
> Erik Johansson wrote:
>
>> How much of the DB load comes from the read only part of the  API, and
>> what if you remove the area limit on the map call?
>
> If I remove the area limit then somebody will do a massive query that
> will suck up all the memory on the machine and everything will die.
>
> That limit is not (primarily) about the cost of gathering the required
> data, though obviously that might become an issue as well, it's about
> the fact that we are holding the whole result set in memory (several
> times over in fact) on the rails server and those servers only have a
> finite amount of memory.

I'm not sure I understand why you do this.
The most expensive call would probably be /map with a bounding-box.
You query all the node, stream them out and keep the nodeIDs
(but not the whole resultset with lat, lon, version, ...) in memory.
Query the ways that use these nodes, keeping their IDs in memory.
Then Query the relations, stream them out and forget about the nodes
and ways while doing that.

The most efficient data-structure to hold IDs in is either an array, bitset or
a hashset, depending on your access-pattern. As the runtime should
be dominated by the database-calls anyway and array or bitset can be used
as it consumes less memory here.

So...why is it that you hold the result-set of the nodes-query in memory
again?

Anyway,  the question was on how much of the  current load is done by
non-writing queries (not just /map). Do we have an answer for that?


Marcus