[Tile-serving] [openstreetmap/osm2pgsql] Drop slim mode tables just after finishing processing pending ways an… (#612)

[Sarah Hoffmann]

lonvia requested changes on this pull request.



> +    fprintf(stderr, "Mid: removing persistent node cache at %s\n", fname);
+
+    // if we don't close the file, *NIX systems keep the inode and its blocks
+    // around
+    if (m_fd >= 0) {
+        close(m_fd);
+        m_fd = -1;
+    }
+
+    // but as the file is mmap'ed, we also have to release the shared_ptr
+    // which deletes the index_t, which unmmaps
+    m_index.reset();
+
+    if (remove_file) {
+        unlink(fname);
+    }

You are duplicating destructor code here. If you just move the three last lines to the end of the destructor, you can get rid of this entire function.

> @@ -32,6 +33,8 @@ class node_persistent_cache
     std::shared_ptr<node_ram_cache> m_ram_cache;
     int m_fd;
     std::unique_ptr<index_t> m_index;
+    bool remove_file;
+    const char *fname;

Coding style: follow the example and prefix members with m_.

> @@ -1072,7 +1072,10 @@ void middle_pgsql_t::pgsql_stop_one(table_desc *table)
 void middle_pgsql_t::stop(void)
 {
     cache.reset();
-    if (out_options->flat_node_cache_enabled) persistent_cache.reset();
+    if (out_options->flat_node_cache_enabled) {
+        persistent_cache->clean_up();
+        persistent_cache.reset();
+    }

I still think that the solution should be generalized, so the benefit is there even without --drop. To achieve that the stop() function needs to be split at this point, the upper part executed before the indexing and the lower part needs to remain where the stop() call was before this change.

And if we do that, then the future vector below can even be merged with the one with the indexing functions in osmdata.cpp below, meaning that we get a much better parallelization.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/612#pullrequestreview-88313231
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20180111/9050bcb1/attachment.html>