<p dir="auto">Flex outputs uses a lot of connections, basically O(number of threads * number of tables). They are mostly idle but need resources on the server. And the PostgreSQL default max number of connections is often not large enough.</p>
<p dir="auto">With these changes the number of connections is much smaller, connections are not open for such long a time.</p>
<ul dir="auto">
<li>Use the same connection for all tables when creating the tables and various other setup tasks</li>
<li>We have one extra connection for the COPYs</li>
<li>Clustering, index creation is done in one connection per thread in the thread pool</li>
</ul>
<p dir="auto">There is probably more we can do here, closing some connections earlier or so. But this is a big step solving most of the problem.</p>

<hr>

<h4>You can view, comment on, or merge this pull request online at:</h4>
<p>  <a href='https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134'>https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134</a></p>

<h4>Commit Summary</h4>
<ul>
  <li><a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/commits/ac6008bdf7d25692b796a371f87e67ef3aac5b93" class="commit-link">ac6008b</a>  Give flex tables a number and use it for prepared statements</li>
  <li><a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/commits/974447fb4e417314177f6f7d3e58cb9833e1204c" class="commit-link">974447f</a>  Refactor: Move storage of db connections from flex tables to output</li>
  <li><a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/commits/ef8f6e50f33b35635c226816c451bfd317300902" class="commit-link">ef8f6e5</a>  Use far fewer connections in flex output</li>
</ul>

<h4 style="display: inline-block">File Changes </h4> <p style="display: inline-block">(<a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files">6 files</a>)</p>
<ul>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-378abe7e8bd6f61501744bcc128b6c65b17b6d68858146870fff281a420cea45">src/flex-lua-table.cpp</a>
    (3)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-1805c9b742d8d030a8d1eb13c8ef3f89183c40f71eecb646e591c7cd0c1d2b24">src/flex-table.cpp</a>
    (85)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-3eae076e672de2c081bf15f0d826d9cfb5415af741a5e24bc8294798aa02a8dc">src/flex-table.hpp</a>
    (31)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-becb223d44aa5a6bb197bf494c1e7e66308350843d927aed3363ce76a1945004">src/output-flex.cpp</a>
    (23)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-785c8da946a79954c6089076f7bf9a585dc6822f989220fa918499e415dd5e46">src/output-flex.hpp</a>
    (5)
  </li>
  <li>
    <strong>M</strong>
    <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134/files#diff-2953d6beaff11241c80c0d723addd63736b9f3715cce5c91ae3de62738daf49b">tests/test-flex-indexes.cpp</a>
    (22)
  </li>
</ul>

<h4>Patch Links:</h4>
<ul>
  <li><a href='https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.patch'>https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.patch</a></li>
  <li><a href='https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.diff'>https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134.diff</a></li>
</ul>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AA6353X6PBDGKM3L3W5NSDLYR6KOLAVCNFSM6AAAAABCY4273OVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTOMJSGM3DAOI">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/AA6353WDLCPFOJDOZDBTKULYR6KOLA5CNFSM6AAAAABCY4273OWGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHH4MF6DE.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><osm2pgsql-dev/osm2pgsql/pull/2134</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134",
"url": "https://github.com/osm2pgsql-dev/osm2pgsql/pull/2134",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>