[OSM-dev] OSM tile server: concurrency of rendering + updating

[Jon Burgess]

On Wed, 2009-10-21 at 13:07 +0200, Frederik Ramm wrote:
> Hi,
> 
>     this is mostly a question to Jon Burgess  I guess but I'm asking on 
> dev because I figure that others might also be interested. On the main 
> OSM tile server, do you apply the diffs while you render tiles at the 
> same time, and have you performed any PostGIS magic to ensure that tile 
> rendering still has priority while diffs are being applied?
> 
> On my setup, I have resorted to using two separate databases (on 
> separate machines) in order to be able to update one while the other 
> renders tiles; doing both at the same time just slowed down tile 
> rendering to a crawl. Now this was not fancy hardware but a reasonably 
> modern quadcore processor and SATA disks - do fat RAM & fast disks make 
> the difference between possible and impossible?

I use a single DB on the tile server but perform the full import into a
new set of tables. The replication diffs are applied into the main
tables roughly once per minute in parallel with any rendering. 

Performing concurrent rendering & importing definitely does slow things
down. I have seen some evidence that running both slows things down to
less than half the speed of running each individually but I have not
tried to quantify it.

During the full planet import I will import the data into a new set of
tables with the renderer accessing the old data. 

A breakdown of the steps I do when performing the bulk update is:
- Download new planet dump
- Stop the diff import
- Drop the tables used by the diffs: nodes, ways, rels.
- Import the new data with a new table prefix (osm2pgsql -p ...)
- Wait many hours for import to complete
- Stop rendering
- Drop old point, line, polygon & roads.
- Rename new tables with the planet_osm_ prefix
- Restart rendering daemon
- Restart diff import but with tile invalidation disabled
- Wait for diffs to catch up (may take a day or two)
- Turn on tile invalidation
- Touch the planet_import_complete flag to force all tiles to re-render

During those steps I also run the coastcheck program to generate an
updated set of coastline shapefiles.

  Jon