[Tile-serving] [openstreetmap/osm2pgsql] Enable Postgres connection polling or query connection limit (#885)

[Paul Norman]

> The default number of processes is normally set to the number of processes available in the system. If you got a number that is larger than 20, it makes me wonder if you have an unusually large system or if something is going wrong with the detection.

The number of connections used is about num tables * num processes, so I'd expect 80 connections for 20 processes, which would fit in the defaults. The limit is based on # of threads, not # of cores, so it's pretty easy to exceed 20 with only two sockets. 

> Independently of that your suggestion to cap the number of processes in the default is still good one. If concurrency gets too high, postgres does not fare too well in my experience. I'd probably go for 15. @pnorman, any opinions?

Postgres' parallelism has gotten better with each version, and I would go much higher than 15. It'd depend on hardware, but I'd generally try 75% of hardware threads, keeping in mind that some stages are single-threaded and will not parallize at all.

The problem that I see is that each process needs a connection for each table instead of one connection for the entire process. This is a real issue with the multi backend, where you might have 30+ tables.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/885#issuecomment-454659366
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190115/3e3402d9/attachment-0001.html>