<p dir="auto">In append mode osm2pgsql has to find all parent ways and relations of changed nodes and parent relations of changed ways. This is done by with one query per object read from the input file which is quite wasteful. This changes the code to only do a single query for all dependencies. The list of all changed ids is COPYied into a temporary table. Then parent objects ids are found and written into another temporary table and then retrieved. (We could this without the extra temporary tables, but this could allow us later to use the content of those tables somewhere else, so I chose to do it this way. Should make a big difference perfomance-wise.)</p>
<p dir="auto">These PR also <span class="issue-keyword tooltipped tooltipped-se" aria-label="This pull request closes issue #423.">fixes</span> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="98484485" data-permission-text="Title is private" data-url="https://github.com/openstreetmap/osm2pgsql/issues/423" data-hovercard-type="issue" data-hovercard-url="/openstreetmap/osm2pgsql/issues/423/hovercard" href="https://github.com/openstreetmap/osm2pgsql/issues/423">#423</a>, the longstanding problem that ways and relations that are in the input file AND parents of changed objects were process twice.</p>
<p dir="auto">The work is split between three commits, one each for nodes, ways, and relations processing.</p>
<p dir="auto">I did some performance measurements, in every one the new code was more efficient. For smaller (minutely) diffs the difference is small, about 30% performance improvements measured over 2000 something diffs. At the other end of the spectrum the performance gain is huge. Updating a planet file with a diff with about two weeks of data was an order of magnitude faster (from >10h down to an hour).</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/openstreetmap/osm2pgsql/pull/1996'>https://github.com/openstreetmap/osm2pgsql/pull/1996</a></p>
<h4>Commit Summary</h4>
<ul>
<li><a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/commits/ea7858abd9db36534a44a9b9bd6623816541881c" class="commit-link">ea7858a</a> Find node dependencies faster by doing a single lookup</li>
<li><a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/commits/fa8336d715e4950cad8db2cedc536b3b1a7a1a5a" class="commit-link">fa8336d</a> Find way dependencies faster by doing a single lookup</li>
<li><a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/commits/36b4fb859dbfda9734df76a347b934c2f685549c" class="commit-link">36b4fb8</a> Make sure not to process relations twice in append mode</li>
</ul>
<h4 style="display: inline-block">File Changes </h4> <p style="display: inline-block">(<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files">9 files</a>)</p>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-3f235baaba072f149d9939d174c0c8326deddc5c92810eeac7aba3d9dfd31ba5">src/dependency-manager.cpp</a>
(80)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-a95858830af94ab23cf5e8c2fbdbc69b1aef032ad960743f20a20c5e977d189f">src/dependency-manager.hpp</a>
(45)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-6237ae72dac71b4b88a541860afdb44f23a47ae252f8236a4522eacd2d6a766f">src/middle-pgsql.cpp</a>
(248)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-658c580d094c7d459c656034c1ad794752b5dbe1e440df0fc026e5396743fd78">src/middle-pgsql.hpp</a>
(13)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-701e90286007c8f36e7825136161bdcc828cd93654be765f06416d40dc43b80a">src/middle.hpp</a>
(16)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-00e919435c3d05ab2a151e1f76fd1ca5e8bfe15d1610cc55e000a515de622f29">src/osmdata.cpp</a>
(22)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-f0af2c86234281cd817df6950c4bdc4a2a12ec6ae2f43b6f0cb9dac69080d3e9">src/pgsql-helper.cpp</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-cb3e1cea5933482434d8769d976b43323d380bc0c65f97cf896ebe48bb8d267e">src/pgsql-helper.hpp</a>
(3)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/1996/files#diff-10e97df69348a578ce413ed39c0d68713777bb2cbc056f481a60610ee447328a">tests/test-middle.cpp</a>
(10)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/1996.patch'>https://github.com/openstreetmap/osm2pgsql/pull/1996.patch</a></li>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/1996.diff'>https://github.com/openstreetmap/osm2pgsql/pull/1996.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/openstreetmap/osm2pgsql/pull/1996">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AA6353Q67VKLKKULDHWT34TXQAHK5ANCNFSM6AAAAAA2JDTVIY">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/AA6353X53HAHTGLVFXV53BTXQAHK5A5CNFSM6AAAAAA2JDTVI2WGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHGW65BYI.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><openstreetmap/osm2pgsql/pull/1996</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/openstreetmap/osm2pgsql/pull/1996",
"url": "https://github.com/openstreetmap/osm2pgsql/pull/1996",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>