[Tilesathome] API Bottleneck

spaetz osm at sspaeth.de
Thu Sep 20 10:58:38 BST 2007


On Thu, Sep 20, 2007 at 08:49:24AM +0100, 80n wrote:

> The current bottleneck for t at h appears to be the throttled API.  

Yes, it definitely is. The t at h server would be able to process way more tilesets than it currently does.

> Hopefully
> this will ease a bit after the NL upload is complete, although there will
> still be plenty of load from the TIGER upload.

I don't think that will ease the situation significantly. I am putting my hopes into the "quadtiling" scheme which TomH plans to implement (see the wiki for some details). I think it will make the API fast enough to be able to handle things again. He could deploy that really quick now but needs to coordinate the few hours of API downtime with others first, AFAIK.

My second hope is the "dirty tiles" requester that bobkare is putting together. It examines a planet-diff and requests all those tiles which have been modified. If we used that centrally, we could only request tiles that have actually been changed, rather than blindly requesting huge areas which probably haven't changed at all. If that is in place, we could throttle other bulk requests completely, until the API becomes responsive again.
 
> I'm wondering if we should consider a mechanism whereby priority 2 requests
> are serviced by the Zappy API.  Currently a lot of the priority 2 requests
> are taking over a week to get processed so the Zappy API would be just as
> good even though it's data is up to a week old.

That sounds like an interesting idea. I'd imagine that we could have priority 3 requests which are served by zappy or something like that.
However, most bulk requests are really done to update areas to the latest data set, so by using planet dump data, that purpose would be somewhat defeated. I'm not saying that's a reason not to do it, but it would probably annoy many, who could then set up their own private queues which would fetch from the current API.

> I would need to make a small change to extend the Zappy API to recognize the
> /api/0.4/map?bbox= style of request, but that is pretty trivial.  There
> would also need to be a change to  tilesGen.pl to tell it where to get the
> data from.

Yes, currently tilesGen.pl doesn't know which priority a request has at all, so it would involve some changes (which would certainly be doable). 

> Perhaps then all pending priority 2 tiles that are unprocessed each
> Wednesday evening could automatically be flagged as being renderable by the
> Zappy API.

sounds definitely interesting. Should we wait a few days to see if tom's quadtile scheme is heping enough already (and using bobkare's dirty tile requester)?

spaetz




More information about the Tilesathome mailing list