[OSM-talk] Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

[Roland Olbricht]

Hi all,

> There is a new release of the Overpass API
> https://dev.overpass-api.de/releases/osm-3s_v0.7.56.6.tar.gz
> that adheres to these rules.

Unfortunately, getting network communication right is hard. In that
version, there is a race condition. For that reason, there is another
new release 0.7.56.7.

You can run that release and keep the database, in particular if an
update based on 0.7.56.4 or 0.7.56.6 has ended with

     gzip: stdin: unexpected end of file
     Reading XML file ...Parse error at line 23105:
     unclosed token

This means that apply_osc_to_db.sh has tried to use an XML file before
fetch_osc.sh has completed to download it. It is harmless, in this case
the automatic emergency stop has applied early enough. Please do the
following (with EXEC_BASE_DIR, DB_DIR and DIFF_DIR appropriately replaced):

# Install new bash scripts over the exisiting broken ones
$ pushd "$EXEC_BASE_DIR"
$ cp ~/osm-3s_v0.7.56.7/bin/fetch_osc.sh bin/
$ cp ~/osm-3s_v0.7.56.7/bin/apply_osc_to_db.sh bin/

# Kill the frozen existing processes and remove the lock file
$ ps -ef | grep apply
roland   19676     1  0 Aug13 ?        00:00:08 bash
bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic

$ ps -ef | grep 19676
roland   19676     1  0 Aug13 ?        00:00:08 bash
bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic
roland   20652  4841  0 04:09 pts/0    00:00:00 grep --color=auto 19676
roland   21629 19676  0 Aug13 ?        00:00:00 ./update_from_dir
--osc-dir=/tmp/osm-3s_update_VUkjcE --version=2020-08-13T20\:33\:03Z
--keep-attic --flush-size=0

$ kill 19676 21629
$ rm /ssd/osm3s/db/osm_base_shadow.lock

$ ps -ef | grep fetch
roland   23675     1  0 Aug13 ?        00:00:06 bash bin/fetch_osc.sh
4146694 https://planet.openstreetmap.org/replication/minute/
/opt/osm3s/diff/
roland   27830  4841  0 04:11 pts/0    00:00:00 grep --color=auto fetch

$ kill 23675

# Restart the diff processes
$ nohup bin/fetch_osc.sh $(cat /ssd/osm3s/db/replicate_id)
'https://planet.openstreetmap.org/replication/minute/' "$DIFF_DIR"
 >>fetch_osc.out &
$ cat "$DB_DIR/replicate_id"
4150150
# Must exist and contain a meaningful diff id, this is a new lock file

$ nohup bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic
 >>apply_osc_to_db.out &
$ sleep 30
$ tail apply_osc_to_db.out
/opt/osm3s/bin /opt/osm3s
Reading XML file ...
# Should show some messages after the "read unclosed token" error
# The first ones are the output of pushd to $DB_DIR/bin and the progress
line "Reading XML file ..." and so on

I'm sorry for the inconvenience and hope that this has fixed all race
conditions. Not yet adapted is "fetch_osc_and_apply.sh". If someone uses
this then please mail me. I will then watch with you whether an improved
version of that script is stable.

Best regards,
Roland