[Tile-serving] [openstreetmap/osm2pgsql] Deleted features after update (#938)

[timautin]

Hello,

I'm setting up an OSM server that I want to keep synced hourly (could be minutely or daily if needed, it doesn't really matters). Everything seems to work (outputs of osmosis & osm2pgsql looks fine, osmosis does download the changes, and osm2pgsql does something to the database). But I have two following problems:
- if I add / edit a feature on OSM iD, my local database does not get the update
- some features are removed (for instance yesterday I lost the USA boundaries polygon, and today France & Spain boundaries are gone as well)

Here's the complete set of command I'm using:

1°) Database import

```
osm2pgsql \
    -d osm \
    --create --slim -G --hstore \
    -C 20000 \
    --number-processes 1 \
    --tag-transform-script ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.lua \
    --style ~/src/map-styles/openstreetmap-carto/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    <path/to/planet-osm.pbf>
```
The process took ~24h, and created a 719GB database (with last week's planet PBF).

2°) Prepare sync (only once)

```
# Set the location
WORKOSM_DIR=~/osm-sync

# Get the PBF file timestamp
TIMESTAMP=$(osmium fileinfo <path/to/planet-osm.pbf> | grep osmosis_replication_timestamp= | cut -c 35-54)

YEAR=$(echo $TIMESTAMP | cut -c 1-4)
MONTH=$(echo $TIMESTAMP | cut -c 6-7)
DAY=$(echo $TIMESTAMP | cut -c 9-10)
HOUR=$(echo $TIMESTAMP | cut -c 12-13)

# Download the initial state.txt
wget "https://replicate-sequences.osm.mazdermind.de/?Y=$YEAR&m=$MONTH&d=$DAY&H=$HOUR&i=00&s=00&stream=hour" -O $WORKOSM_DIR/state.txt

# Create the config file
rm -f $WORKOSM_DIR/configuration.txt
osmosis --read-replication-interval-init workingDirectory=$WORKOSM_DIR

# Update the config file
sed -i 's!http:!https:!' $WORKOSM_DIR/configuration.txt
sed -i 's!minute!hour!' $WORKOSM_DIR/configuration.txt
```

3°) Continue to sync (this is called every hour with a CRON task)

```
# Set some variables
WORKOSM_DIR=~/osm-sync
OSM_CARTO_DIR=<path-to-openstreetmap-carto>
export EXPIRE_FILE_NAME=expire-list.txt

# Download changes
osmosis \
    --read-replication-interval workingDirectory=$WORKOSM_DIR \
    --simplify-change \
    --write-xml-change $WORKOSM_DIR/tmp/osmChange.xml

# Apply the changes
osm2pgsql \
    --database gis \
    --append --slim --multi-geometry --hstore \
    --cache 3000 \
    --tag-transform-script $OSM_CARTO_DIR/openstreetmap-carto.lua \
    --style $OSM_CARTO_DIR/openstreetmap-carto.style \
    --flat-nodes /ssd/flat-nodes/osm.cache \
    --expire-tiles 10-20 \
    --expire-output $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME \
    --input-reader xml \
    $WORKOSM_DIR/tmp/osmChange.xml

# Expire tiles in [10-15]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=10 \
    --max-zoom=15 \
    --touch-from=10 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME

# Delete tiles in [16-20]
render_expired \
    --tile-dir /hdd/tile-cache/mod_tile \
    --min-zoom=16 \
    --max-zoom=20 \
    --delete-from=16 \
    --socket /var/run/renderd.sock < $WORKOSM_DIR/tmp/$EXPIRE_FILE_NAME
```

Did I miss something, or could the feature deletion be a bug?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/938
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190710/737d1e3d/attachment.html>