[OSM-dev] Tirex vs Renderd

[Jochen Topf]

On Sat, Apr 17, 2010 at 12:09:52PM +0100, Kai Krueger wrote:
> On 04/17/2010 11:29 AM, Frederik Ramm wrote:
> > Kai,
> >
> > Kai Krueger wrote:
> >> This is less of a question though regarding using renderd vs tirex but
> >> more of a question of where the future lies and thus for which to
> >> develop new features. It would be a shame and rather inefficient to
> >> put in effort to develop features for something that turns out to be a
> >> dead end
> >
> > What particular feature did you want to work on?
> 
> It was somewhat intended as a more general question than a specific 
> feature, but what sparked the question was the potential need to extend 
> the rendering to support a lot of styles (200+) as is currently being 
> discussed on the wikimedia maps list. At 
> http://cassini.toolserver.org/tile-browse/ there are renderings for all 
> the 278 languages wikipedia supports. With renderd each of the rendering 
> threads has a copy of all the mapnik styles. As each style takes up 
> about 5 - 6 Mb, this results in 278*8*5.5 = 12Gb of ram. Not directly 
> desirable. So thoughts exist to implement some form of on demand loading 
> and unloading of styles, as it is unlikely that all those styles will be 
> needed all the time and/or share memory accross rendering 
> threads/processes. So the question somewhat was where to do this 
> potential development. But as it is only thoughts at the moment it will 
> probably depend on who ever gets to actually implementing this. Hence my 
> more general question of which of the two seems more "recomended/future 
> proof" for potential developments like that.

Having 278 styles that only differ in a few places where the different
languages are involved certainly seems wasteful. We have already thought
about the on-demand-loading of styles and it is probably something we
want to implement for Tirex. There is a different use case that also needs
that: When you are working on and testing a new style, you don't want to
restart your renderer all the time to reload styles. Thats a different
problem, but it would need a similiar solution, ie load styles on use.

But in your case it might not solve the problem. Depending on how often
tiles are asked for for the different languages, you might still end up
with either most of the styles in memory or otherwise need to constantly
reload the styles which takes time.

There are many ways we could approach this: One would be to implement
something in Mapnik so that it can allow variables in style files. Then
you would only have one style file for all languages and somehow feed
the language in separately.

Another approach would be to render the map in one step and the labels
in a second step and merge them. You still had one style file for each
language, but they would be much smaller, because they contain only the
labels. This merging could be done in the renderer. Bascially it would
call Mapnik twice and merge the images. Or maybe we can modify Mapnik
so that it can not only render into a new empty bitmap, but also into
an existing one. Something like this.

Most of these approaches are probably easier with Tirex, because we
separated the queue management from the rendering. And, as I said in my
other mail, I am currently working on making the rendering backend more
configurable and flexible. The idea is that you can configure for any
map, which rendering backend to call with which parameters. So it should
be easy to experiment with a new backend.

Jochen
-- 
Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298