[OSM-dev] Carto-style rendering and PostgreSQL parallel Worker

[Sarah Hoffmann]

On Wed, May 05, 2021 at 10:03:02PM +0300, Darafei "Komяpa" Praliaskouski wrote:
> Parallel workers should not kick in on the rendering setups as parallelism
> there is driven by having metatiles rendered in parallel. Postgres
> parallelism is for long running analytic queries, not fast "get me
> everything in a small box" ones.

You are right in theory. In practise, the introduction of parallel
workers and JIT has introduced a lot of headaches for OSM setups.

The underlying problem is that table statistics tend to be way off.
Consequently, the query planner would compute much higher execution
times than what the query actually takes.

Just to be clear, way off means: by two or three
orders of magnitude. That never posed a problem because the statistics
were right relative to each other and the query planer would still
choose the right indexes etc. But since we have JIT and parallel
workers, the bad statistics mean that the query planner decides to
use them way to early.

We force JIT and parallel execution to off for osm2pgsql and
Nominatim these days because we know that all queries are short
and won't profit anyway.

I don't know what the situation is with rendering but switching JIT
and parallel workers off sounds like the more cautious approach.

Sarah

> 
> On Wed, May 5, 2021 at 9:59 PM Stephan Knauss <osm at stephans-server.de>
> wrote:
> 
> > Hello,
> >
> > what is your experience with PostgreSQL parallel workers on a map
> > rendering setup?
> >
> > On the psql mailing list there was a recent discussion regarding whether
> > the default value for the planner to decide for parallel workers is too
> > low.
> >
> > I decided to give it a try and disabled parallel workers completely by
> > setting
> >
> > max_parallel_workers_per_gather = 0
> >
> > I do monitor the rendering throughput on Z10-20 and load value.
> > I see no larger change in rendering throughput, but a significant
> > reduction on the load value, as there is no longer a second process to
> > be scheduled.
> >
> > What is your experience with parallel workers on a rendering setup?
> > Does it bring benefit for you? Did you change the cost factors?
> >
> > I am rendering an extract and assume that a large part of the database
> > is cached in RAM, other parts are on fast nvme storage. So the overhead
> > in cost to set up a parallel worker might be a point to consider.
> >
> > I marked on the chart the time at which I disabled parallel workers:
> >
> > https://i.postimg.cc/xdLTHXrY/parallel-worker-load-rendering-rate.png
> >
> > Stephan
> >
> > _______________________________________________
> > dev mailing list
> > dev at openstreetmap.org
> > https://lists.openstreetmap.org/listinfo/dev
> >
> 
> 
> -- 
> Darafei "Komяpa" Praliaskouski
> OSM BY Team - http://openstreetmap.by/

> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> https://lists.openstreetmap.org/listinfo/dev