[Tile-serving] Pre-Rendering tiles for all of Germany

[Paul Norman]

On 12/27/2017 1:41 PM, Josef Schugt wrote:
> Hi there,
>
> a few months ago, our company's then system administrator who among 
> other responsibilities had the task of bulk pre-generating 
> custom-design tiles had the opportunity to start working at the German 
> Federal Office for Information Security which he quite understandably 
> took. When I write 'tiles' I actually mean it; it is not metatiles - 
> our customer doesn't want the tiles to be generated on the fly.
>
> Since then I have been more or less successful in little by little 
> understanding and improving the system used and now wonder if the 
> fundamental approach actually is the right one.

It isn't. As a very rough ballpark, this will be 50M total tiles being 
250GB.

> ISSUE 1: HOW DATA IS IMPORTED
>
> The region for which tiles are needed is Germany in its current 
> borders. The first issue that arises is the data import (we use data 
> from download.geofabrik.de, please see 
> http://download.geofabrik.de/europe/germany.html).
>
> The machine we use memory-wise comes to its limits when generating 
> indexes after importing germany-latest.osm.pbf. Sounds odd at first 
> but there is considerably more OSM data for Germany than for the 
> Russian Federation while Russia has about 50 times the area that 
> Germany has - about one twelvth of the amount of data available for 
> the whole planet!

Configure PostgreSQL with suitable maintenance_work_mem and use an 
appropriate --cache value for osm2pgsql. You could import successfully 
with far less RAM than you need for acceptable performance. If you're 
very RAM constrained, use --disable-parallel-indexing.

> Is it possible to import the data federal-state-wise? I know that I 
> can use osmconvert to make sure that there are no collisions by using 
> something like
>
> [...]
> osm2pgsql ... --append ... second_region_cleaned.pbf

Don't try using append like this. It will be very slow, and you'll need 
to rebuild all the indexes when you're done anyways.

> ISSUE 2: HOW RENDERING IS PERFORMED
>
> The rendering takes place using a modified version of 
> https://github.com/openstreetmap/mapnik-stylesheets/blob/master/generate_tiles.py 
> where the modification is using a couple of regions that cover all of 
> Germany and some output that allows to have a web page for keeping an 
> eye on the progress: http://tileserver.benndorf.de/tile/progress/
>
> The data used for the marker (click on it, it has additional 
> information - note that most up to date Firefox is fubar and won't 
> display it) is updated every 1000 tiles and the page automatically 
> refreshes every 5 minutes. Each region is rendered at zoom levels 10 
> thorough 18, then the program moves on to the next region (the rows 
> are processed from North to South, within each row the regions are 
> processed from West to East).
>
> We (i.e. my boss and I) are under the impression that for such bulk 
> generation it should be faster to frist generate all metatiles and 
> from these generate all actual tiles. Or, simply put: use mod_tile and 
> simply fetch each and every tile abusing Apache as the de facto 
> rendering engine. However, the machine we'd like to use for this 
> purpose has mod_tile operational for on-demand rendering of other 
> tiles and I am not sure if it is a good idea to have a second instance 
> of mod_tile doing lots of rendering in parallel.

Yes. This will also have much better label placement. mod_tile supports 
multiple map styles, so you won't need multiple instances running in 
parallel. There will still be a performance hit, but this will happen 
with any method.

Instead of mod_tile, you could look at other methods like mapproxy. I 
believe it has better support for rendering regions that aren't squares.


> Any substantial feedback is appreciated. If you have comments on the 
> granularity of the grid used for rendering all of Germany, feel free 
> to share them as well. I still have no feeling for what the optimal 
> size of a region for rendering is. To my understanding it should not 
> be too large because then the whole data used no longer fits into 
> memory and not too small because then there are way too many features 
> that need to be fetched many times - say autobahns that go on for 
> hundreds of kilometres.

You want to just give it the entire list of tiles and have it work 
through them. The issues you're describing don't make sense, because the 
metatiles are the same size regardless of how much you render at once.