[Tile-serving] Fedora 18 ansible playbook to have mod_tile, Tirex, Nominatim, Mapnik

[Kai Krueger]

On 03/28/2013 08:33 AM, Bernard Fouché wrote:
> Hi List,
>
> as mentioned on dev at openstreetmap.org, here is the complete steps I 
> had to follow to setup a server providing tiles and [reverse]geocoding 
> on a target running Fedora 18, in the event that it can be useful for 
> someone else.
There are also a set of puppet scripts to set up a mod_tile / renderd 
based tile server at https://gerrit.wikimedia.org/r/#/c/36222/ It does 
not include the nominatim part though. If I am not mistaken, the OSM 
admins also have a set of chef recepies to get these services up and 
running, but I don't think they are public yet.
>
> It is incomplete because it references files like a virtual-host file 
> of your own, copies of some projects like Tirex or Nominatim, etc. but 
> at least if someone has to take a similar road (as of March 28th 
> 2013), then here is a guide that can help to solve the problem (I 
> guess ansible wizards and/or OSM specialists can do much better). 
> Files permissions are also probably badly handled, as postgresql users .
>
> This script is also posted here also in the event that it can show the 
> real needs:
>
> -  to simplify such a setup
> - to handle version numbering for the components involved.
>
> Ansible (http://ansible.cc) syntax used here is pretty obvious, each 
> 'step' is described following 'name:', each 'action' describes what to 
> do. ansible has built in modules (like 'command', 'yum', copy, etc) 
> but I'm not a specialist so I often use 'shell' and then type pure 
> shell commands.
I have added some comments inline below.
>
> #
> # This playbook installs openstreetmap / mapnik on a target
> #
> ---
> - hosts: ... the name of the target to setup....
> sudo: yes
>
> vars:
> scripts: ../../scripts
> conf: ../../conf
> rpm: ../../rpm
> nominatim: ../../nominatim
> osmosis: ../../osmosis
> postgis: postgis-2.0.3-1.fc19.x86_64.rpm
> # Monaco is perfect for testing; small, fast to process and full of 
> details
> openstreetmap_planet: monaco-latest.osm.bz2
>
> tasks:
>
> - name: Check we have latest packages (mod_tile compilation may fail 
> otherwise with org pkgs from install CD)
> action: command yum -y update
> tags:
> - osm
>
> [...]
>
> #
> # Postgreqsl and tiles are stored into '/data', a specific partition. 
> This means that to benefecit from
> # this you should unmount/mount '/data' (or whatever location you use)
> #
> - name: Have '/data' mounted with 'noatime' to speed up postgresql / 
> tiles reading
> action: shell sed -i -e 's/dataext4defaults/dataext4noatime/' /etc/fstab
> tags:
> - osm
>
> [...]
> #
> # Mod_tile setup is the virtual host file loaded by httpd
> #
> - name: Have httpd to include a virtualhost file
> action: lineinfile dest=/etc/httpd/conf/httpd.conf create=no 
> insertafter=EOF regexp="Include conf/virtual-hosts.conf" line="Include 
> conf/virtual-hosts.conf" state=present
> tags:
> - osm
>
> - name: Copy our virtual host file
> action: copy src=$conf/virtual-hosts.conf 
> dest=/etc/httpd/conf/virtual-hosts.conf owner=root mode=644
> tags:
> - osm
>
> #
> # It is unclear yet if this file is really needed or not. There is 
> 'LoadTileConfigFile /etc/renderd.conf' in my
> # virtual host file so I keep it at the moment.
>
This depends. For simple configurations and if you are using tirex, this 
is not necessary. Instead of using the LoadTileConfigFile directive in 
Apache, you can also use the AddTileConfig or AddTileMimeConfig 
directives in the Apache config to load each tileset individual. For 
more complex configurations you do however need the file based 
configuration mechanisms.
>
> #
> - name: Copy renderd.conf which is referenced by virtual-hosts.conf
> action: copy src=$conf/renderd.conf dest=/etc/renderd.conf owner=root 
> mode=644
> tags:
> - osm
>
> [...]
>
> # user 'www-data' unused at the moment (?)
> - name: Create 'www-data' user for DB, accept error if already created
> action: shell sudo -u postgres createuser -S -D -R www-data
> ignore_errors: True
> tags:
> - osm
>
> #
> # This is the 'usual' script importing for instance 
> /usr/share/pgsql/contrib/postgis-64.sql
> # and granting privileges to apache (don't know if this is really needed)
> #
> - name: Copy script that setup rights for pgsql
> action: copy src=$scripts/psqlconf.sh dest=/tmp/psqlconf.sh mode=555 
> owner=postgres
> tags:
> - osm
>
> - name: Run script that setup rights for pgsql
> action: shell sudo -u postgres /tmp/psqlconf.sh
> tags:
> - osm
>
> [...]
> - name: Checkout Mapnik data from SVN (we'll use only data from it not 
> code since we have the mapnik RPM)
> action: shell chdir=/data/osm svn co 
> http://svn.openstreetmap.org/applications/rendering/mapnik
> tags:
> - osm
>
You don't need to manually download all of the coastline files.
The 
https://github.com/openstreetmap/mapnik-stylesheets/blob/master/get-coastlines.sh
script does this for you. I.e. downloads the data and extract it, 
removing a bunch of steps from your playbook.
<https://github.com/openstreetmap/mapnik-stylesheets/blob/master/get-coastlines.sh> 

>
> - name: Test if coastline data (1) is already available
> action: shell test -f /data/osm/world_boundaries-spherical.tgz
> ignore_errors: True
> register: world_boundaries_spherical
> tags:
> - osm
>
> - name: Download coastline data (1)
> action: shell chdir=/data/osm wget 
> http://tile.openstreetmap.org/world_boundaries-spherical.tgz
> when_failed: $world_boundaries_spherical
> tags:
> - osm
>
> - name: Test if coastline data (2) is already available
> action: shell test -f /data/osm/processed_p.tar.bz2
> ignore_errors: True
> register: processed_pl
> tags:
> - osm
>
> - name: Download coastline data (2)
> action: shell chdir=/data/osm wget 
> http://tile.openstreetmap.org/processed_p.tar.bz2
> when_failed: $processed_pl
> tags:
> - osm
>
> - name: Test if coastline data (3) is already available
> action: shell test -f /data/osm/shoreline_300.tar.bz2
> ignore_errors: True
> register: shoreline_300
> tags:
> - osm
>
> - name: Download coastline data (3)
> action: shell chdir=/data/osm wget 
> http://tile.openstreetmap.org/shoreline_300.tar.bz2
> when_failed: $shoreline_300
> tags:
> - osm
>
> - name: Test if file required since 2010 (1) is already available
> action: shell test -f /data/osm/ne_10m_populated_places.zip
> ignore_errors: True
> register: 10m_populated_places
> tags:
> - osm
>
> - name: Supplementary file required since 2010 (1)
> action: shell chdir=/data/osm wget 
> http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places.zip
> when_failed: $10m_populated_places
> tags:
> - osm
>
> - name: Test if file required since 2010 (2) is already available
> action: shell test -f /data/osm/110m_cultural.zip
> ignore_errors: True
> register: 110m_admin_0_boundary_lines
> tags:
> - osm
>
> - name: Supplementary file required since 2010 (2)
> action: shell chdir=/data/osm wget 
> http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m_cultural.zip
> when_failed: $110m_admin_0_boundary_lines
> tags:
> - osm
>
> - name: Have /data/osm/mapnik to hold OSM data and scripts
> action: file path=/data/osm state=directory owner=postgres 
> group=postgres mode=755
> tags:
> - osm
>
> #
> # Should find some test to avoid uncompressing these archives each time
> #
> - name: Extract world_boundaries-spherical.tgz
> action: shell chdir=/data/osm/mapnik tar xvzf 
> /data/osm/world_boundaries-spherical.tgz
> tags:
> - osm
>
> - name: Extract processed_p.tar.bz2
> action: shell chdir=/data/osm/mapnik tar xvjf 
> /data/osm/processed_p.tar.bz2 -C world_boundaries
> tags:
> - osm
>
> - name: Extract shoreline_300.tar.bz2
> action: shell chdir=/data/osm/mapnik tar xvjf 
> /data/osm/shoreline_300.tar.bz2 -C world_boundaries
> tags:
> - osm
>
> - name: Extract ne_10m_populated_places.zip
> action: shell chdir=/data/osm/mapnik unzip -o 
> /data/osm/ne_10m_populated_places.zip -d world_boundaries
> tags:
> - osm
>
> - name: Extract 110m_cultural.zip
> action: shell chdir=/data/osm/mapnik unzip -o 
> /data/osm/110m_cultural.zip -d world_boundaries
> tags:
> - osm
>
> #
> # I just don't remember where I found these :-( Anyway the Fedora 
> mapnik RPM don't have any
> # font bundled with them (Fedora policy: don't bundle fonts, have 
> separated font packages)
> # It is probably possible to find them in a better way than keeping a 
> copy.
>
In Debian/Ubuntu there are separate packages for the fonts in the 
repository. "ttf-dejavu" and "unifont". I presume Fedora will have 
something similar
>
> #
> - name: Copy fonts for mapnik (where did I get them :-( ?)
> action: copy src=$rpm/mapnikfonts.tgz dest=/tmp
> tags:
> - osm
>
> - name: Extract mapnik fonts
> action: shell chdir=/data/osm/mapnik tar zxf /tmp/mapnikfonts.tgz
> tags:
> - osm
>
> #
> # Would it be possible to combine both mapnik/nominatim run of 
> osm2pgsql in a single one?
> #
> - name: Import data for Mapnik with OSM2PGSQL
> action: shell sudo -u importer 
> /data/nominatim/Nominatim/osm2pgsql/osm2pgsql --slim -S 
> /data/nominatim/Nominatim/osm2pgsql/default.style -C 12000 
> /data/osm/$openstreetmap_planet
> tags:
> - osm
>
> #
> # Probably unneeded, I had to setup this to use some software, I don't 
> remember which one
>
Apxs is the apache build system and is needed to build mod_tile. Instead 
of symlinking things, you could have also just used ./configure 
--with-apxs=/bin/apxs
I have also now improved the configure scripts to look for apxs in more 
places automatically, so it should now just find it automatically with 
out any extra things to do.
>
> #
> - name: make /usr/bin/apxs2 a link to /bin/apxs
> action: shell ln -fs /bin/apxs /usr/bin/apxs2
> tags:
> - osm
>
> #
> # Using rev 29418, no official version numbering for mod_tile...
> #
> - name: Get mod_tile source code (only for mod_tile, not renderd)
> action: shell chdir=/data svn co 
> http://svn.openstreetmap.org/applications/utils/mod_tile/
> tags:
> - osm
>
> - name: Autogen mod_tile.
> action: shell chdir=/data/mod_tile ./autogen.sh
> tags:
> - osm
>
> - name: Configure mod_tile.
> action: shell chdir=/data/mod_tile ./configure
> tags:
> - osm
>
> - name: Make mod_tile.
> action: shell chdir=/data/mod_tile make
> tags:
> - osm
>
> #
> # It is unclear if these two steps are required or if the second one 
> is enough
>
No, they are both necessary. Well, actually, if you are using tirex 
instead of renderd, then you probably only need the make 
install-mod_tile, but I haven't tried that.
>
> #
> - name: Install mod_tile (step 1/3)
> action: shell chdir=/data/mod_tile make install
> tags:
> - osm
>
> - name: Install mod_tile (step 2/3)
> action: shell chdir=/data/mod_tile make install-mod_tile
> tags:
> - osm
>
> - name: Install mod_tile (step 3/3)
> action: shell chdir=/data/mod_tile ldconfig
> tags:
> - osm
>
> - name: Have httpd to load module mod_tile
> action: copy src=$conf/mod_tile.conf dest=/etc/httpd/conf.d/mod_tile.conf
> tags:
> - osm
>
> - name: Provide legacy SQL function to GIS database
> action: shell sudo -u postgres psql -d gis < 
> /usr/share/pgsql/contrib/postgis-2.0/legacy.sql
> tags:
> - osm
>
I don't thin generate_xml.py has been necessary for the last couple of 
years. The configuration is now done in the template files in the inc 
sub directory.
>
> - name: Generate osm.xml file
> action: shell chdir=/data/osm/mapnik ./generate_xml.py --dbname gis 
> --user importer --accept-none
> tags:
> - osm
>
If you use renderd instead of tirex, you can skip most of these steps, 
as you have already built (and installed) everything necessary there and 
it can share the configuration file with mod_tile.
>
> - name: add a Tirex user
> action: user name=tirex state=present
> tags:
> - osm
>
> #
> # Ours is 29418 with two changes: one to remove '-l[agg]' from the 
> output of 'mapnik-config',
> # the second to avoid including mapnik/expression.php .
> #
> - name: bring our copy of Tirex
> action: copy src=$rpm/tirex.tgz dest=/data owner=tirex mode=644
> tags:
> - osm
>
> - name: Untar Tirex archive
> action: shell chdir=/data tar zxf tirex.tgz
> tags:
> - osm
>
> - name: Give /data/tirex to user tirex
> action: shell chown -R tirex:tirex /data/tirex
> tags:
> - osm
>
> - name: Make Tirex
> action: shell sudo -u tirex chdir=/data/tirex make
> tags:
> - osm
>
> - name: Install Tirex
> action: shell chdir=/data/tirex make install
> tags:
> - osm
>
> - name: Tirex install sets things we don't need in /etc/tirex, remove them
> action: shell chdir=/etc/tirex/renderer rm -rf *.conf mapserver test wms
> tags:
> - osm
>
> #
> # The copied configuration only uses mapnik renderer, for a map named
> # default, having tiles in /data/tiles/default
> #
> - name: Copy our Tirex configuration
> action: copy src=$conf/tirex.conf.tar dest=/data owner=root
> tags:
> - osm
>
> - name: Store Tirex configuration
> action: shell chdir=/ tar xf /data/tirex.conf.tar
> tags:
> - osm
>
You change the name to something else in the tile layer specification 
file. For mod_tile and renderd that is in the renderd.conf file It is 
the name in the square brackets. For Tirex there is also a config file 
per tile layer where you can change this name
>
> - name: Make /data/tiles/default (default is the name of the map, 
> dunno where to change this)
> action: file path=/data/tiles/default state=directory owner=root mode=777
> tags:
> - osm
>
You don't need to create symlinks to /var/lib/mod_tile to move the tile 
directory somewhere else. You can simply change the tile dir in the 
configuration files. For mod_tile, you can set the "ModTileTileDir 
/data/tiles" directive in the site definition file. You can also do it 
on a tile layer by tile layer basis in the renderd.conf file.
>
> - name: Remove symlink /var/lib/mod_tile
> action: shell rm -f /var/lib/mod_tile
> tags:
> - osm
>
> - name: Make symlink from /var/lib/mod_tile to /data/tiles
> action: shell ln -fs /data/tiles /var/lib/mod_tile
> tags:
> - osm
>
> - name: Have directory /var/lib/tirex
> action: file path=/var/lib/tirex state=directory owner=tirex mode=755
> tags:
> - osm
>
> - name: Make symlink from /var/lib/mod_tile to /data/tiles
> action: shell ln -fs /data/tiles /var/lib/mod_tile
> tags:
> - osm
>
> [...]
>
>
>     Bernard
>
>
>
>
> _______________________________________________
> Tile-serving mailing list
> Tile-serving at openstreetmap.org
> http://lists.openstreetmap.org/listinfo/tile-serving