[OSM-dev] Tirex - a renderd alternative

[Frederik Ramm]

Hi,

    a few days ago I committed a first working version of "Tirex" to our 
svn (under applications/utils). Tirex is basically an alternative for 
renderd, it does the same things, just differently.

We (Geofabrik) had to roll out a tile server for one of our clients (a 
company named Enaikoon from Berlin - they do vehicle tracking and other 
telematics stuff). They wanted some functionality that would have 
required extensive changes to renderd, and we decided to write something
new instead. Enaikoon likes OSM and open source, so the result is a 
fully open-source OSM project just as everything else which you can try 
out, run, and improve if you want.

The main difference between the classic renderd and Tirex is that in 
Tirex, the actual rendering and the queue management are strictly 
separate. You have one daemon which only manages requests and queues and
priorities and all that, and you have another set of daemons that do 
nothing but tile rendering. Communication between these is done using 
UDP, with a very simple protocol where you have one key=value pair per 
line. Tirex does not use multithreading at all - everything is either 
multiplexing inside one process, or multiprocessing.

The bit that handles the queue is called the Tirex master. It is written
in Perl to allow for easy manipulation of policy decisions. The Tirex 
master supports any number of rendering queues (here called buckets) 
with a high degree of configurability. For example, if you know that you
have 16 backend rendering processes, then you can configure the master 
to use no more than 10 of these for low-priority requests (and only do 
so below a certain load) so that you always retain spare rendering 
capacity for interactive requests. Rendering requests can also have an 
expire time so that if the queue manager could not get them rendered 
within a specified time they are discarded.

The bit that does the actual rendering is the "Tirex rendering daemon" 
or "Tirex renderd". It is written in C++ but does very little - read 
rendering request, call Mapnik, create Metatile. The plan is to support 
other rendering backends as well. You will usually have a number of 
these running (and they are automatically re-started if they die e.g. 
due to a problem in the Mapnik library).

In addition there's a number of helper programs, for example a nifty 
status console that lets you see what your queues are doing, munin 
plugins, and a tool to mass-enqueue requests ("everything in the 
following bbox from z6 to z10 but only if the queue is not too long") 
and other helpful stuff.

Further documentation is in SVN. A debian subdirectory is provided, so
on reasonably current Debian and Ubuntu distributions you should be able
to simply do a "debuild" to create a number of .deb packages.

We do not have a replacement for mod_tile yet but are planning to have
one in the future. At the moment you can simply use the existing 
mod_tile, unchanged, with Tirex. You can also use all of the existing 
utilities (render_list etc.) with Tirex. Tirex uses the same meta tiles 
and the same directory structure, making it easy to run it as a drop-in 
replacement for renderd if desired.

We don't expect Tirex to quickly replace renderd - renderd is well 
established and does what it is expected to do. But we believe that the 
more modular structure of Tirex makes it easier for developers to play 
around and plug in their own little bits and pieces (just as it was for us).

We do have some plans for Tirex ourselves and will surely continue to 
contribute, but we do not plan to keep this as "our own", and we will 
work with the OSM svn version in the future just as everyone else. Tirex
is in production on tile.openstreetmap.de since two weeks ago, and seems
to work well.

Feedback is welcome.

There's a Wiki page named "Tirex" with about the same content as this.

Bye
Frederik