[Tile-serving] [openstreetmap/osm2pgsql] Fix multistage processing (take 2) (#1215)

[Sarah Hoffmann]

@lonvia commented on this pull request.



>  {
-    if (m_dependency_manager->has_pending()) {
-        multithreaded_processor proc{m_conninfo, m_mid, m_outs,
-                                     (std::size_t)m_num_procs};
+    if (!m_dependency_manager->has_pending()) {
+        return;
+    }

This only excludes stage 1b, not stage 1c.

> +    multithreaded_processor proc{m_conninfo, m_mid, m_outs,
+                                 (std::size_t)m_num_procs};
+
+    // stage 1b processing
+    proc.process_ways(m_dependency_manager->get_pending_way_ids());
+    proc.process_relations(m_dependency_manager->get_pending_relation_ids());
+    proc.merge_expire_trees();
+
+    // If none of the outputs wants to do stage 2 processing, we can skip
+    // stage 1c.
+    if (std::none_of(m_outs.cbegin(), m_outs.cend(),
+                     [](std::shared_ptr<output_t> const &out) {
+                         return out->has_stage2_processing();
+                     })) {
+        return;
+    }

If there is no stage2 processing, get_marked_way_ids()  simply is empty. So the loop below seems enough.

> +        return;
+    }
+
+    if (ltype != LUA_TTABLE) {
+        throw std::runtime_error{
+            "Table returned from mark_relation_members() contains 'ways' "
+            "field, but it isn't an array table"};
+    }
+
+    // Iterate over the 'ways' table to get all ids...
+    lua_pushnil(lua_state());
+    while (lua_next(lua_state(), -2) != 0) {
+        if (!lua_isnumber(lua_state(), -2)) {
+            throw std::runtime_error{
+                "Table returned from mark_relation_members() contains 'ways' "
+                "field, but it isn't an array table"};

...array table of numbers

> @@ -1185,6 +1264,7 @@ void output_flex_t::way_delete(osmid_t osm_id)
 
 void output_flex_t::relation_delete(osmid_t osm_id)
 {
+    mark_relation_members(osm_id);

Do we call the output_delete before deleting from middle? I suspect not.

>  
 You are expected to define one or more of the following functions:
 
-* `osm2pgsql.process_node()`: Called for each node.
-* `osm2pgsql.process_way()`: Called for each way.
-* `osm2pgsql.process_relation()`: Called for each relation.
+* `osm2pgsql.process_node()`: Called for each new or changed node.
+* `osm2pgsql.process_way()`: Called for each new or changed way.
+* `osm2pgsql.process_relation()`: Called for each new or changed relation.
+* `osm2pgsql.mark_relation_members()`: Called for each deleted or added

I'm still not quite happy with the name. The function doesn't mark anything, it returns members to be reprocessed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/1215#pullrequestreview-432179962
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20200617/22755b4f/attachment-0001.htm>