[OSM-dev] OSM API Performance (Was: OSM Database Infrastructure)

[Tom Hughes]

In message <20080119234103.GA1098 at alta.metacarta.com>
          Christopher Schmidt <crschmidt at metacarta.com> wrote:

> I'm working on creating interfaces that display OSM data as you drag the
> map around. The interface requests data from OSM "on the fly" -- so
> every time the map stops moving, the old features are removed, and
> a new request is sent out.

I know you are, although I have to admit to being somewhat concerned
as to whether we are ever likely to have the infrastructure necessary
to support such activity on any significant scale.

> Note that all of these times are *before* the download starts -- once it
> starts, it takes practically no time at all (as I would expect, since I
> think all the work is done before th e download starts).

Correct. All the information is gathered in memory and formatted
as XML and then sent to the client.

> I don't know what makes this process slow. By comparison, taking a
> similarly sized area from osm2pgsql via FeatureServer's PostGIS interface
> is typically only approximately 1s lag time. The wild variation in times
> -- which doesn't seem to be tied to things being in local cache, as one
> 3s request can be followed immediately after by a 9s request for the
> exact same area -- makes me wonder if perhaps this is not tied to the
> data being requested, but the 'backup' on the rails daemons... in which
> case, possibly just throwing more hardware at the problem would fix it?

Well unless we have infinite resources we are never going to be
able to guarantee instant response. I wouldn't even say it is even
a target as it isn't the primary use of the API at least as things
stand.

> In the past, I think that we had talked about the fact that the /map
> request is resource intensive compared to most of the other ruby calls,
> and that some of it is memory + CPU intensive and might be a small chunk
> worth improving performance in. Given that sometimes the requests are
> returning reasonably quickly, this may be premature optimization if al
> lthe wait time is waiting for an available rails daemon.

It is not especially CPU intensive. It is I/O and memory intensive
but there isn't much we can do about that (well there is, but it
probably isn't an answer people will want to hear).

The limit on the number of rails daemons is more or less directly
related to memory consumption anyway, so saying that optimising the
memory use won't help if the problem is waiting for a rails daemon
is not really accurate.

The biggest limitations on API response times are, in no particular
order (because the three are intimately linked), ruby, rails, and
memory usage.

> How many rails daemons are there for the non-t at h API? Is there a way to
> see how long the queue to get to a rails daemon is? I guess even more
> importantly than either of those, is it possible that this explanation
> is the case?

There are nine daemons for non-t at h API calls - three on www and
six on db2. There is no way to see the queue for those (you can
see the overall queue on munin but not the queue for a specific
class of daemon).

I suspect it's just as likely that any slowness you're seeing is
due to hitting a daemon that is running a machine that is swapping
due to excessive memory usage though, a situation that normally
corrects itself fairly quickly.

> It's naturally true that if there is a limit to the number of clients
> talking to the db at once, there is a peak utilization which the
> database can't get past. However, it's possible that even with a
> completely capable database setup (which I agree exists now), there are
> other things limiting performance of the site and development of tools
> that could exist if the API was faster (which then leads to a vicious
> cycle where performance goes down again... :))

There are still issues with the way we query the database for the
map call, notably the need to fetch a list of nodes and then feed
a (potentially very large) list of node IDs back to the database
to find the matching ways.

In principle it should be possible to find the ways which have
nodes in an area without that, but due to MySQL being utterly
rubbish at optimising subqueries I have yet to come up with a
workable query to do so.

If we could do that then in principle we could stream data to the
client, thus avoiding the need for large memory usage. I say in
principle because in practice rails makes it very hard, if not
completely impossible, to do so.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/