<p>This pull request contains the changes announced in <a href="https://github.com/openstreetmap/osm2pgsql/issues/709" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/709" data-id="213343500" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#709</a>.</p>
<p>This pull request is the first part of a rewrite of the tile expiry. It is a port of the tile expiry of <a href="https://github.com/Nakaner/Cerepso">my own</a> PostgreSQL import tool to osm2pgsql which I wrote as part of my master thesis. I needed an importer with a tile expiry which handles relations in much different way than osm2pgsql does and the tile expiry had to be more exact.</p>
<p>Each tile is represented by a 64-bit integer (bits of Y and X index interleaved). The set only contains the tiles at the highest zoom level where the list of expired tiles should be generated. All lower requested zoom levels are calculated during writing the tile IDs to the output file by shifting the bits of the tile IDs.</p>
<p>I did a performance test with an import of the Europe extract by Geofabrik to ensure that it has no negative impact on the performance. Time and memory consumption was measured using <code>/usr/bin/time -v</code>. The input data was located on a hard disk, flat nodes file and the database was located on SSDs.</p>
<ol>
<li>Check out the <code>master</code> branch.</li>
<li>Import using <code>osm2pgsql --merc --slim --flat-nodes /ssd/michael-flatnodes/michael-flat-nodes.cache -C 40000 --style ../default.style --unlogged --number-processes 2 --database gis europe-170217.osm.pbf</code> took 24 hours and 35 minutes and needed 26,765,608 kB. During these 24 hours the machine was busy producing a subset of Geofabrik's extracts offered on download.geofabrik.de for internal purposes, i.e. don't take these numbers serious.</li>
<li>Save a copy of the flat nodes file.</li>
<li>Shut down the PostgreSQL database.</li>
<li>Copy the tablespace of the database <code>gis</code> to a backup location.</li>
<li>Start the database.</li>
<li>Apply a diff using <code>osm2pgsql -d gis --slim --merc -S ../default.style --flat-nodes /ssd/michael-flatnodes/michael-flat-nodes.cache --append --number-processes 2 -C 25000 -o master-433.log --expire-bbox-size 20000 -e 10-16 433.osc.gz</code>. (56 minutes 28 seconds, 9,143,644 kB RAM)</li>
<li>Check out branch <code>expiry-as-set</code> and build osm2pgsql. The original import could be reused because the differences between both branches were small.</li>
<li>Restore the backup of the flat nodes file.</li>
<li>Shut down the PostgreSQL database.</li>
<li>Restore the tablespace of the database <code>gis</code> from the backup.</li>
<li>Start the database.</li>
<li>Apply a diff using <code>osm2pgsql -d gis --slim --merc -S ../default.style --flat-nodes /ssd/michael-flatnodes/michael-flat-nodes.cache --append --number-processes 2 -C 25000 -o master-433.log --expire-bbox-size 20000 -e 10-16 433.osc.gz</code>. (57 minutes 34 seconds, 9,389,628 kB RAM)</li>
</ol>
<p>The cronjob which produces the OSM planet extracts day by day on this machine did not run during step 8 and 14. The result is within the measurement accuracy. The code line responsible for writing the expiry file was removed in both test cases to ensure that writing larger output files (see <a href="https://github.com/openstreetmap/osm2pgsql/issues/709" class="issue-link js-issue-link" data-url="https://github.com/openstreetmap/osm2pgsql/issues/709" data-id="213343500" data-error-text="Failed to load issue title" data-permission-text="Issue title is private">#709</a>) does not influence the measurements.</p>
<p>If you request tile expiry for smaller zoom levels (e.g. 8 to 12), both branches consume about 500 MB RAM.</p>
<p>This is the first part of a large rewrite of the tile expiry. If you are interested in the ongoing work, have a look at the branch <a href="https://github.com/Nakaner/osm2pgsql/tree/tile-selection-rewrite"><code>tile-selection-rewrite</code></a> at my fork of this repository. That branch will rewrite the selection of the tiles which will be marked as expired.</p>
<hr>
<h4>You can view, comment on, or merge this pull request online at:</h4>
<p> <a href='https://github.com/openstreetmap/osm2pgsql/pull/747'>https://github.com/openstreetmap/osm2pgsql/pull/747</a></p>
<h4>Commit Summary</h4>
<ul>
<li>Rewrite backend of tile expiry</li>
<li>remove unnecessary code, write an expired tile only once (bugfix)</li>
<li>renaming quadtree and qt to quadkey</li>
<li>various minor corrections based on feedback</li>
<li>replace unordered sets by a sorted vector at output of expiry list</li>
<li>minor performance improvements for tile expiry</li>
<li>add two test cases for the tile expiry</li>
<li>fix signed-to-unsigned warnings, use maxzoom=0 for "no tile expiry"</li>
<li>check if zoom levels for tile expiry are smaller than 32</li>
</ul>
<h4>File Changes</h4>
<ul>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/747/files#diff-0">expire-tiles.cpp</a>
(247)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/747/files#diff-1">expire-tiles.hpp</a>
(196)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/747/files#diff-2">options.cpp</a>
(15)
</li>
<li>
<strong>M</strong>
<a href="https://github.com/openstreetmap/osm2pgsql/pull/747/files#diff-3">tests/test-expire-tiles.cpp</a>
(430)
</li>
</ul>
<h4>Patch Links:</h4>
<ul>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/747.patch'>https://github.com/openstreetmap/osm2pgsql/pull/747.patch</a></li>
<li><a href='https://github.com/openstreetmap/osm2pgsql/pull/747.diff'>https://github.com/openstreetmap/osm2pgsql/pull/747.diff</a></li>
</ul>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/openstreetmap/osm2pgsql/pull/747">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AD2-7nkzIKXnz9TDxoAJmYK5BcSB_Dd6ks5rx3cfgaJpZM4NDLff">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AD2-7qd5ZwjaQEfsRSh_hFfN5MaeEKKtks5rx3cfgaJpZM4NDLff.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/openstreetmap/osm2pgsql/pull/747"></link>
<meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/openstreetmap/osm2pgsql","title":"openstreetmap/osm2pgsql","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/openstreetmap/osm2pgsql"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Rewrite tile expiry (management of expired tiles) (#747)"}],"action":{"name":"View Pull Request","url":"https://github.com/openstreetmap/osm2pgsql/pull/747"}}}</script>