[Tile-serving] [osm2pgsql-dev/osm2pgsql] Significantly reduce number of db connection in flex output (PR #2134)

[Jochen Topf]

Flex outputs uses a lot of connections, basically O(number of threads * number of tables). They are mostly idle but need resources on the server. And the PostgreSQL default max number of connections is often not large enough.

With these changes the number of connections is much smaller, connections are not open for such long a time.

* Use the same connection for all tables when creating the tables and various other setup tasks
* We have one extra connection for the COPYs
* Clustering, index creation is done in one connection per thread in the thread pool

There is probably more we can do here, closing some connections earlier or so. But this is a big step solving most of the problem.
You can view, comment on, or merge this pull request online at:

  https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134

-- Commit Summary --

  * Give flex tables a number and use it for prepared statements
  * Refactor: Move storage of db connections from flex tables to output
  * Use far fewer connections in flex output

-- File Changes --

    M src/flex-lua-table.cpp (3)
    M src/flex-table.cpp (85)
    M src/flex-table.hpp (31)
    M src/output-flex.cpp (23)
    M src/output-flex.hpp (5)
    M tests/test-flex-indexes.cpp (22)

-- Patch Links --

https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.patch
https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.diff

-- 
Reply to this email directly or view it on GitHub:
https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134
You are receiving this because you are subscribed to this thread.

Message ID: <osm2pgsql-dev/osm2pgsql/pull/2134 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20240204/19b549e8/attachment.htm>