[OSM-dev] Extract update forgets relation

[Stephan Knauss]

Hello,

I run an extract database and render maps there.

After a recent change the newly added parts of this relation are no 
longer rendered.

https://thaimap.osm-tools.org/?zoom=15&lat=20.37243&lon=99.47782&layers=BT

I checked the database and the complete relation disappeared.

I suspect some glitch in the way I update the extract.

Instead of importing all ways and relations, as only nodes can be 
filtered against coordinates, and by cron cleaning up later as I did 
before (see 
https://wiki.openstreetmap.org/wiki/User:Stephankn/knowledgebase#Cleanup_of_ways_outside_the_bounding_box)

I now use the trim_osc.py script advertised on switch2osm.

What step might trigger osm2pgsql to drop a relation? What has to happen 
on the input data?

I faced a similar issue before. There I noticed that the relation 
modification caused a "gap" in the outer path of the relation. 
Reordering fixed the problem.

In this case the ways are all pointing into the same direction and are 
contiguous (eg way 140845137).

Unfortunately the changeset is quite large, so I have trouble 
understanding what happened to the way which formed the boundary before. 
The addition of the new way segments and modification of the relation 
are within the same changeset.
https://www.openstreetmap.org/changeset/102940425

Is anyone seeing something obvious which could explain this? What are 
the recommendation to debug it?

I thought of importing a test-db just before the relevant change-set and 
then checking the individual steps of the import of the offending osc.

This sounds unfortunately quite time-consuming.
Any other ideas?

Where could it fails? One idea is that the trim_osc fails for some 
reason to include the newly added ways. The ways in question are very in 
the middle of the bounding polygon. So the I know that it does some 
queries to check whether a relation should be included. Could this fail?

./trim_osc.py -d gis --host postgis --use osm -p osm-tools-sea.poly -v 
-z --password ${TEMP_DIR}/change.osc.gz ${TEMP_DIR}/change-trim.osc.gz

What would be the consequence further down the pipe? If the relation is 
no longer in OSC, would this anyhow affect the relation handling later 
by osm2pgsql? Or does the deletion of the relation indicate that it was 
inside the trimmed osc file as otherwise osm2pgsql would not have 
considered it?

Later in the pipe osm2pgsql is updating the database. Besides removing 
the relation it also seems to correctly create an expiry list, as the 
gap was promptly removed from the rendering.

What condition could trigger osm2pgsql to remove the element from the 
database?

osm2pgsql --multi-geometry --slim --append --hstore --username osm 
--host postgis --database gis --expire-tiles 3-17 --expire-output 
${TEMP_DIR}/expire.list --style 
openstreetmap-carto/openstreetmap-carto.style --tag-transform-script 
openstreetmap-carto/openstreetmap-carto.lua ${TEMP_DIR}/change-trim.osc.gz

Versions are quite recent, osm2pgsql at 1.4.1, carto for the lua at 
5.3.1, trim-osc is latest with 
https://raw.githubusercontent.com/Zverik/regional/43ed342f1ad10df0acf08f572f31b124ef72aef9/trim_osc.py

Database is PostgreSQL 13.2 with Postgis 3.1.

Stephan