[Tile-serving] Pre-Rendering tiles for all of Germany

[Marcos Dione]

On Wed, Dec 27, 2017 at 10:41:47PM +0100, Josef Schugt wrote:
> The machine we use memory-wise comes to its limits when generating indexes
> after importing germany-latest.osm.pbf. 

    please consider adding more ram. it's cheap and it will save you a
lot of headaches. also, please share the parameters you're using when
invoking osm2pgsql. for reference, I can import the whole Europe with
16GiB of RAM... also, use SSDs, not HHDs. this one is not so cheap, but
Germany should fit several times in a 250GiB SSD. I can import Europe in
such a disk.

    also, think about using an initial import + daily updates, not
reimporting every time.

> ISSUE 2: HOW RENDERING IS PERFORMED
> 
> The rendering takes place using a modified version of https://github.com/openstreetmap/mapnik-stylesheets/blob/master/generate_tiles.py
> where the modification is using a couple of regions that cover all of
> Germany and some output that allows to have a web page for keeping an eye on
> the progress: http://tileserver.benndorf.de/tile/progress/
> 
> The data used for the marker (click on it, it has additional information -
> note that most up to date Firefox is fubar and won't display it) is updated
> every 1000 tiles and the page automatically refreshes every 5 minutes. Each
> region is rendered at zoom levels 10 thorough 18, then the program moves on
> to the next region (the rows are processed from North to South, within each
> row the regions are processed from West to East).
> 
> We (i.e. my boss and I) are under the impression that for such bulk
> generation it should be faster to frist generate all metatiles and from
> these generate all actual tiles. Or, simply put: use mod_tile and simply
> fetch each and every tile abusing Apache as the de facto rendering engine.
> However, the machine we'd like to use for this purpose has mod_tile
> operational for on-demand rendering of other tiles and I am not sure if it
> is a good idea to have a second instance of mod_tile doing lots of rendering
> in parallel.

    at the beginning of your mail you said:

> our customer doesn't want the tiles to be generated on the fly.

    so are you, or aren't you generating tiles on the fly?

    if not, I also have a heavily modified version of
generate_metatiles.py[1] that can handle rendering Europe from ZL0 to 14 in
a weekend. it uses a couple of tricks like:

* rendering 8x8 metatiles, then splitting them in the individual ones.
* rendering depth first, so data used for tile x,y in ZLn is hot in cache
  for the 4 'child' tiles in ZL n+1, and so on ecursively. I don't have 
  strong numbers showing this helps, but rendering Europe in a weekend is  
  good enough for me. 
* stop rendering when the tile is pure water. this saves a lot of useless 
  work, except in the boundary betweenZL7 and ZL8, where ferry routes 
  appear. currently the 'empty sea' tile contents (varies with your style) 
  is harcoded.

> Any substantial feedback is appreciated. If you have comments on the
> granularity of the grid used for rendering all of Germany, feel free to
> share them as well. I still have no feeling for what the optimal size of a
> region for rendering is. To my understanding it should not be too large
> because then the whole data used no longer fits into memory and not too
> small because then there are way too many features that need to be fetched
> many times - say autobahns that go on for hundreds of kilometres.

    I don't understand what you mean here...

--
[1] https://github.com/StyXman/elevation/blob/master/generate_tiles.py
-- 
(Not so) Random fortune:
The technology industry sees itself as in rebellion against corporate
America: not corrupt, not buttoned-up, not empty. In fact, a tech company
can be as corrupt, soulless, and empty as any corporation, but being
unprofessional helps us maintain the belief that we are somehow different
from Wall Street.
	    -- Shanley Kane