[OSM-dev] Two technology related questions

[Peter Wendorff]

Hi Sandor,
I'll try to answer your questions between your original post.

Am 30.01.2014 11:23, schrieb Sandor Seres:
> I am not sure am I in the right forum to ask the two tiling related
> questions here. If not, I do hope someone could redirect me to the right
> place/forum. 
> 
> As I understand the OSM raster-tiles serving system is one of the
> fundamental OSM services. Even the front page map (the SlippyMap and its
> layers) is based on this service as well as many private mapping systems.
> Further I understand that these tiles are pre-generated, edge aligned colour
> raster images in PNG format of a uniform dimension and created per zoom
> levels. At the same time, the OSM source data is (implicitly and apart from
> the attributes) in a vector format. 
That's correct
> So, at some stage the mentioned service
> must perform a vector-to-raster transformation and here is my first
> dilemma/question:
The software doing that transformation in case of the tiles shown on
osm.org is called Mapnik, see www.mapnik.org
> 
> 1. Are you rasterizing geometry/vector objects first for a larger area and
> then tiling this raster image, or the contrary? First tiling the
> geometry/vector objects and so rasterizing these vector tiles?
Something like that.
Mapnik usually is configured to render a block of e.g. 5x5 tiles as one
big image, the so called meta-tile and slices the result down to
individual tiles.
To avoid clipping effects neigbouring meta-tiles overlap and the
overlapping parts are ignored.

On the other hand the vector data is kind of tiled too by using
appropriate database indices and - initially - storing the vector data
in a projection which is suitable to easily clip to the area of interest.

> Next, I think I understand how the maintenance of this raster tiles database
> happens. When a mapper uploads a change all related tiles are marked for
> re-rendering.
Yes, but to make sure you understand: There are two databases and one
set of files involved:
The Mapper uploads to the api-database.
A process produces updates of that api-database in minutely chunk files
(so approximately once a minute there's a new osm-change-file)
At the rendering server a second process fetches the osm-change-files
and parses it. While parsing it does:
- the projection (as I wrote above) suitable for rendering
- insert/update/delete each object into the database
- mark affected tiles as dirty for each changed/inserted/deleted object.

> So, at an appropriate time the marked tiles are re-rendered
> and the users have an impression (more or less) that the updates happen
> on-the-fly. 
Basically yes. Mapnik runs as a service with a rendering queue (or
better more than one rendering queue). Tiles not marked as dirty don't
have to be rerendered and are sent to the mappers client on request.

Dirty tiles are rerendered depending on their zoomlevel. The
low-zoom-levels (z0..5 afair) are rendered usually independent of tile
requests as they use a lot of data for the same image size and therefore
rendering takes too much time (and of course they would be dirty very
often as they span across a big area of the world).

Higher zoom levels may be rendered in a render queue too if there's
spare time, but may be rendered with higher priority on demand, too.

I could not find some more details about this marking process.
> So, my second dilemma/question is:
> 
> 2. When an object (given by a relation) has changed, do you mark all the
> related/coverage tiles for re-rendering or only those affected by the
> change?
As far as I know all related/covereage tiles are marked as dirty, but
the low zoom levels are not rerendered on request. So if I am up to date
there is no decision based on attributes (e.g. zoom level 2 is not
affected because there's only a highway-tag which changed and highways
aren't shown on z2).

> Just to be a bit more precise regarding the second question. Assume we have
> an area object, for example a larger lake with islands, given by a relation.
> Assume, a local editor will refine the object by changing just a short
> section of the outer polygon, or/and just to insert a new small missing
> island. The whole object is covered by hundreds of tiles while the actual
> change is covered by only a few.
It's rerendered completely as the whole lake is in the database as one
object, independent of the internal structure of osm objects it is
derived from.

regards
Peter