[Tile-serving] [openstreetmap/osm2pgsql] Fix multistage processing (#1208)

[Sarah Hoffmann]

@lonvia commented on this pull request.



> @@ -22,8 +22,9 @@ tables.routes = osm2pgsql.define_relation_table('routes', {
     { column = 'tags', type = 'hstore' },
 })
 
--- This will be used to store lists of relation ids queryable by way id
-by_way_id = {}
+-- This will be used to store information about relations queryable by member
+-- way id

Explain that this is a table of tables and why. (If this is because relations may appear twice, didn't we exclude this now?)

> @@ -247,6 +246,19 @@ it is not, use stage 1 to store information about OSM objects you will need
 in stage 2 in some global variable. In stage 2 you can read this information
 again and use it to decide where and how to store the data in the database.
 
+Processing in two stages can add quite a bit of overhead. Because this feature
+is new, there isn't much operational experience with it. So be a bit careful
+when you are experimenting and watch memory and disk space consumption and
+any extra time you are using. Keep in mind that:
+
+* All data stored in stage 1 for use in stage 2 in your Lua script will use
+  main memory.
+* Keeping track of way ids marked in stage 1 needs some memory.
+* To do the extra processing in stage 2, time is needed to get objects out
+  of the object store and re-process them.
+* Osm2pgsql will create an id index on all way tables to look up ways that are
+  re-processed.

wasn't that to 'delete ways that are reprocessed' ?

>  
-        proc.process_ways(m_dependency_manager->get_pending_way_ids());
+    // stage 1c processing
+    for (auto &out : m_outs) {
+        if (out->has_stage1c_pending()) {

This if seems overkill. get_stage1c_way_ids() returns an empty array per default anyway.

>  
-        proc.process_ways(m_dependency_manager->get_pending_way_ids());
+    // stage 1c processing
+    for (auto &out : m_outs) {
+        if (out->has_stage1c_pending()) {
+            for (auto const id : out->get_stage1c_way_ids()) {
+                m_dependency_manager->way_changed(id);
+            }
+        }
+    }
+
+    if (m_dependency_manager->has_pending()) {

At this point, we should ignore `add_rows()`. Does this still happen?

> @@ -821,14 +829,7 @@ int output_flex_t::table_add_row()
             throw std::runtime_error{
                 "Trying to add relation to table '{}'"_format(table.name())};
         }
-        if (m_in_stage2) {
-            delete_from_table(&table_connection, osmium::item_type::relation,
-                              m_context_relation->id());
-        }

where did the delete go?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/1208#pullrequestreview-428349527
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20200613/ac1bde59/attachment.htm>