[Tile-serving] [openstreetmap/osm2pgsql] Drop slim mode tables just after finishing processing pending ways an… (#612)

[Marcos Dione]

StyXman commented on this pull request.



> @@ -1072,7 +1072,10 @@ void middle_pgsql_t::pgsql_stop_one(table_desc *table)
 void middle_pgsql_t::stop(void)
 {
     cache.reset();
-    if (out_options->flat_node_cache_enabled) persistent_cache.reset();
+    if (out_options->flat_node_cache_enabled) {
+        persistent_cache->clean_up();
+        persistent_cache.reset();
+    }

>From what I understand, there are 5 modes of import:

1. Normal mode, where intermediate data is created in RAM. It's not clear to me from the doc if this mode is updatable or not, but I guess the answer is yes.
2. `--slim` mode, where that data is written in the database. This mode is updatable.
3. `--slim` + `--flat-nodes`, where par of the data if written in a spaces file and `mmap()`'ed, so no db queries. This mode is also updatable.
4. `--slim --drop` and
5. `--slim --flat-nodes --drop`, were the intermediate data is erased. With this flag, the database is not upgradable anymore.

There are two problems with `--slim --drop`:

* The flat nodes file, if used (mode 5), is not erased. This PR fixes this.
* The intermediate data is removed at the end, even when they're not longer used way before that moment (mainly, just before indexing). This PR fixes this only for `--flat-nodes`, mode 5.

So what you mean is that we should do the second item for both modes 5 and 3. I think it makes sense, and it seems that what you mention about splitting `stop()` is actually making this slower to run: I'm getting 500s+ with this patch vs 230s+ without for importing Argentina in my machine.

I'll give the `stop()` splitting a try tomorrow, but you will have to explain a little bit more about the last paragraph. Can I drop by any IRC channel to discuss it? I think I'll be faster than just ping-pong through GitHub.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/612#discussion_r161099672
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20180111/1564a602/attachment.html>