[OSM-dev] all things to all people

Tom Hughes tom at compton.nu
Tue Apr 22 14:48:36 BST 2008


In message <D6890768-EFA5-4190-AD3D-380D4BE5D715 at asklater.com>
        SteveC <steve at asklater.com> wrote:

> but do we really have the resources to serve arbitrary data to people?  
> I've noticed the tile server taking a nose dive (and api calls  
> actuall) in speed recently  and I thought it was the bandwidth from  
> various locations. But, it may be that the export magic is partially  
> happening on that box?

How recent is recently? Presumably since the weekend?

Export of XML data is just a redirect to the API so is nothing
that didn't exist already and uses the same resources.

Export of mapnik images is done by redirecting to a script on the
tile server, but the munin graphs show no sign of any increased
load since it was deployed (or indeed of any significant load aside
from when planet is being loaded).

Export of osmarender images is done by redirecting to the MapOf call
on the t at h server, which already existed. Obviously it may get used
more now though. There are some speed issues with that, and with t at h
in general I believe. Those are thought to be down to disk I/O issues
and I believe work is ongoing to get new disks there.

> Anecdotally I've heard of a few people hammering the export tab for  
> data. This is exactly why we don't ship WMS and WFS. It's why tiles  
> are good and limited bounding box requests from the map call are good.  
> We will never satisfy the arbitrary data requests for the export tab  
> or WMS etc as there will always be more wanted. Adding an arbitrary  
> output format just adds another arbitrary format on to it, making it  
> more computationally intensive.

Since early Sunday morning when the logs rolled over, the tile
server has served 2002 mapnik export requests, or about one every
two minutes on average. Those requests came from 747 unique addresses
so a bit over 2 per IP on average.

Obviously in fact there are lots that have only done 1 request and
some that have done a dozen or two.

Basically I'm working on the theory of building it, putting it out
and seeing what happens - if abuse proves to be a problem then we have
to move it or implement resource limits. Obviously it will not be as
high a priority service as other things.

> Really, the existing boxes should be throwing out as many tiles and  
> API calls as possible. We should buy a new box for export and it  
> shouldn't be a priority in terms of memory, disk, uptime etc.

It's a nice idea - one major problem is that different formats need
different base resources that are in different places.

We could easily setup another server with a copy of the postgis
database and run mapnik export from it if necessary.

Export of XML data relies on the main database so is more of a
problem.

Likewise export of osmarender images, which works by stitching
together the existing tiles so needs access to the rendered tiles.

Tom

-- 
Tom Hughes (tom at compton.nu)
http://www.compton.nu/




More information about the dev mailing list