[OSM-dev] Scaling Overpass API

mmd mmd.osm at gmail.com
Fri Jul 20 15:26:57 UTC 2018


Am 20.07.2018 um 08:47 schrieb Leon Boehmer:
> 
> I wish to run overpass api in a production environment. However I need
> to make it more scalable. i.e need to make it handle a large amount of
> requests to the dispatcher. I was wondering if anyone has had any
> success in scaling just the dispatchers. I.e. I want one instance of DB
> and updater and then I want to run like 3-4 simultaneous dispatchers. Is
> this possible should I try this? Have people done it before?


Unlike the name suggests, the dispatcher is more of a lock manager to
coordinate read and write access to the different db files on disk. It
doesn't really make sense to scale this component, and a lock file would
even prevent you from doing so.

I somewhat wonder how you came to the conclusion that the dispatcher is
a bottleneck worth scaling up. In my tests [1], I processed >= 2000
requests/s and still the dispatcher isn't near saturation.

Scaling can be naturally achieved by defining more CGI processes on your
web server, i.e. running multiple instances of the interpreter binary.
Also, you should revisit, why you need to make so many requests. You
could probably easily bundle them into larger requests to make the whole
process much faster.


[1] https://github.com/mmd-osm/Overpass-API/issues/2




More information about the dev mailing list