<p>(not ready yet, I want CI to check compiling on Windows and Mac)</p>
<p><em>This is a DB-breaking change.</em> DBs imported with older code can no longer be updated with this code and there will be <strong>no migration</strong> to the new format.</p>
<h2>Unprojected coordinates in node caches</h2>
<p>Reprojection of coordinates into the tile coordinate system is delayed until the point when nodes are retrieved from the middle caches (see in particular the <code>nodes_get_list()</code> functions). This has a few advantages:</p>
<ul>
<li>OSM coordinates can be saved directly, no need for fix-point magic on reprojected coordinates, which avoids a few rounding errors.</li>
<li>no more custom scale factor needed for custom projections (fixes <a href="https://github.com/openstreetmap/osm2pgsql/issues/121" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/121" data-id="27710742" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#121</a>)</li>
<li>no need for floating-point implementation of caches (supersedes <a href="https://github.com/openstreetmap/osm2pgsql/pull/357" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/357" data-id="80672792" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#357</a>)</li>
<li>avoid expensive floating point conversation until last minute</li>
</ul>
<p>The main drawback is that some coordinates now may need to be reprojected multiple times. This is for example the case for nodes where roads cross each other. OTOH unused nodes are no longer converted.</p>
<h2>Osmium Dense File Array</h2>
<p>The custom flatnode file implementation has been replaces with osmium's dense file array index. Its implementation uses mmap, avoiding custom block management. You can also use osmium-based tools to inspect/update the file outside osm2pgsql.</p>
<p>Furthermore, the dense file array should be thread-save for reading. The 4th commit in this PR removes the parts where the file is reopened and uses one flatnode store object throughout the import. (fixes <a href="https://github.com/openstreetmap/osm2pgsql/issues/96" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/96" data-id="21367414" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#96</a>)</p>
<p>There are a couple of additional formatting changes created by git-clang-format.</p>
<p>Resolves <a href="https://github.com/openstreetmap/osm2pgsql/issues/542" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/542" data-id="136922284" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#542</a>.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/openstreetmap/osm2pgsql/pull/668'>https://github.com/openstreetmap/osm2pgsql/pull/668</a></p>
<h4>Commit Summary</h4>
<ul>
<li>store unprojected coordinates in node caches</li>
<li>remove scale option</li>
<li>replace flatnode store with osmium dense file array</li>
<li>allow concurrent reads on flatnode file</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-0">geometry-processor.cpp</a>
(12)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-1">geometry-processor.hpp</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-2">middle-pgsql.cpp</a>
(125)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-3">middle-pgsql.hpp</a>
(10)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-4">middle-ram.cpp</a>
(37)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-5">middle-ram.hpp</a>
(5)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-6">middle.hpp</a>
(11)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-7">node-persistent-cache.cpp</a>
(654)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-8">node-persistent-cache.hpp</a>
(101)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-9">node-ram-cache.cpp</a>
(356)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-10">node-ram-cache.hpp</a>
(81)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-11">options.cpp</a>
(44)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-12">options.hpp</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-13">osmdata.cpp</a>
(20)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-14">output-gazetteer.cpp</a>
(9)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-15">output-gazetteer.hpp</a>
(11)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-16">output-multi.cpp</a>
(106)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-17">output-multi.hpp</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-18">output-null.cpp</a>
(8)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-19">output-null.hpp</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-20">output-pgsql.cpp</a>
(25)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-21">output-pgsql.hpp</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-22">output.hpp</a>
(4)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-23">tests/middle-tests.cpp</a>
(17)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-24">tests/mockups.hpp</a>
(16)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-25">tests/regression-test.py</a>
(36)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-26">tests/test-middle-flat.cpp</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-27">tests/test-middle-pgsql.cpp</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-28">tests/test-middle-ram.cpp</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-29">tests/test-output-pgsql-area.cpp</a>
(1)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-30">tests/test-output-pgsql.cpp</a>
(7)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-31">tests/test-parse-diff.cpp</a>
(6)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/668/files#diff-32">tests/test-parse-xml2.cpp</a>
(5)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/668.patch'>https://github.com/openstreetmap/osm2pgsql/pull/668.patch</a></li>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/668.diff'>https://github.com/openstreetmap/osm2pgsql/pull/668.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/openstreetmap/osm2pgsql/pull/668">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AD2-7g7vgDm_ylciTBEKUj3BUbNRaSUMks5rPCTxgaJpZM4LbKgQ">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AD2-7trZNmZzQg4n590u5wRJ-xqJrxWQks5rPCTxgaJpZM4LbKgQ.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/openstreetmap/osm2pgsql/pull/668"></link>
<meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/openstreetmap/osm2pgsql","title":"openstreetmap/osm2pgsql","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/openstreetmap/osm2pgsql"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Save unprojected coordinates in node caches and use osmium dense file array (#668)"}],"action":{"name":"View Pull Request","url":"https://github.com/openstreetmap/osm2pgsql/pull/668"}}}</script>