[Tile-serving] [openstreetmap/osm2pgsql] Fix multistage processing (take 2) (#1215)

[Jochen Topf]

(This is a replacement for #1208.)

The multi-stage processing in the flex output has some problems when
updating data, for instance when relations are deleted after their
member ways use their tags.

This commit reorganizes how the multi-stage processing is done to
address these problems.

1. The way "marking" of objects is done is now different. Instead of
   doing this in the process_* Lua functions a new Lua function
   "mark_relation_members()" is introduced. This will be called not only
   for all new relations but also for deleted relations, or when a relation
   changed, for the old relation. This is needed so that we also mark and
   then re-create way entries in the database that used to depend on a
   parent relations tags, but don't do that any more. This function
   must return the way ids that need to be re-processed in stage 2.
2. We remove the Lua mark() function. Instead the return value of
   "mark_relation_members()" is used to mark way ids. Marking of nodes
   and relations isn't supported. It was never possible to mark nodes
   anyway. And there is no use case I am aware of currently that needs
   marking relations. Both can be reintroduced later when we have a
   better idea how to handle them. For the time being we concentrate on
   the important use case where member ways of relations are handled
   specially.
3. This introduces a new processing stage 1c:
    * stage 1a: Read input file and process all objects in it.
    * stage 1b: Process dependent objects of objects from 1a (ie
      changed nodes trigger changes in ways with those nodes, changes
      in all objects potentially trigger changes in parent relations).
    * stage 1c: Process dependent relations of objects marked during
      stage 1a/1b (this one is new).
    * stage 2: Process objects marked in stage 1a or 1b.
4. New Lua helper function osm2pgsql.way_member_ids() that returns
   the ids of all way members of the specified relation. This is often
   needed in mark_relation_members(). If you need more complex processing,
   for instance use the member roles to decide which ways need stage 2
   processing, you can still write your own loop.
You can view, comment on, or merge this pull request online at:

  https://github.com/openstreetmap/osm2pgsql/pull/1215

-- Commit Summary --

  * Fix multistage processing

-- File Changes --

    M docs/flex.md (89)
    M docs/lua-lib.md (16)
    M flex-config/route-relations.lua (68)
    M src/init.lua (10)
    M src/middle-pgsql.cpp (2)
    M src/osmdata.cpp (41)
    M src/osmdata.hpp (4)
    M src/output-flex.cpp (183)
    M src/output-flex.hpp (20)
    M src/output.hpp (11)
    M tests/data/test_output_flex_extra.lua (40)
    M tests/test-output-flex-extra.cpp (131)

-- Patch Links --

https://github.com/openstreetmap/osm2pgsql/pull/1215.patch
https://github.com/openstreetmap/osm2pgsql/pull/1215.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/1215
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20200616/e334bc94/attachment.htm>