[Tile-serving] [openstreetmap/osm2pgsql] Segmentation Fault during Nominatim daily update (#1014)

[Arvind Kartik]

Hey @mmd-osm , that is all I got for the `>bt` command

@lonvia :
#1018 did fix the update without crashing but is extremely slow.

Here's the stats from subsequent sequences for example:
```
Currently at sequence 2631 (25/11/2019 23:59:01) - 1 indexed
/srv/nominatim/.local/bin/pyosmium-get-changes --server http://planet.openstreetmap.org/replication/day -o /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc -s 30 -I 2631
/srv/nominatim/Nominatim-3.4.0/build/osm2pgsql/osm2pgsql -klas --number-processes 1 -C 2000 -O gazetteer -S /srv/nominatim/Nominatim-3.4.0/settings/import-full.style -d nominatim -P 5432 /srv/nominatim/Nominatim-3.4.0/b
uild/osmosischange.osc
osm2pgsql version 0.95.0-dev (64 bit id space)
Using projection SRS 4326 (Latlong)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc
Using XML parser.
Processing: Node(2997k 4.6k/s) Way(506k 0.68k/s) Relation(10320 86.00/s)  parse time: 1521s
Node stats: total(2997356), max(7012720590) in 651s
Way stats: total(506834), max(749949585) in 749s
Relation stats: total(10476), max(10340741) in 120s
..
.
Currently at sequence 2632 (26/11/2019 23:58:59) - 1 indexed
/srv/nominatim/.local/bin/pyosmium-get-changes --server http://planet.openstreetmap.org/replication/day -o /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc -s 30 -I 2632
/srv/nominatim/Nominatim-3.4.0/build/osm2pgsql/osm2pgsql -klas --number-processes 1 -C 2000 -O gazetteer -S /srv/nominatim/Nominatim-3.4.0/settings/import-full.style -d nominatim -P 5432 /srv/nominatim/Nominatim-3.4.0/b
uild/osmosischange.osc
osm2pgsql version 0.95.0-dev (64 bit id space)
Using projection SRS 4326 (Latlong)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc
Using XML parser.
Processing: Node(3415k 4.6k/s) Way(524k 0.70k/s) Relation(10300 89.57/s)  parse time: 1611s
Node stats: total(3415536), max(7015777818) in 741s
Way stats: total(524727), max(750312974) in 755s
Relation stats: total(10634), max(10345049) in 115s
..
.
Currently at sequence 2633 (27/11/2019 23:59:01) - 1 indexed
/srv/nominatim/.local/bin/pyosmium-get-changes --server http://planet.openstreetmap.org/replication/day -o /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc -s 30 -I 2633
/srv/nominatim/Nominatim-3.4.0/build/osm2pgsql/osm2pgsql -klas --number-processes 1 -C 2000 -O gazetteer -S /srv/nominatim/Nominatim-3.4.0/settings/import-full.style -d nominatim -P 5432 /srv/nominatim/Nominatim-3.4.0/b
uild/osmosischange.osc
osm2pgsql version 0.95.0-dev (64 bit id space)
Using projection SRS 4326 (Latlong)
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=2000MB, maxblocks=32000*65536, allocation method=11
Mid: pgsql, cache=2000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Reading in file: /srv/nominatim/Nominatim-3.4.0/build/osmosischange.osc
Using XML parser.
Processing: Node(2897k 4.1k/s) Way(444k 0.67k/s) Relation(9970 71.73/s)  parse time: 1505s
Node stats: total(2897514), max(7018312834) in 699s
Way stats: total(444942), max(750611635) in 667s
Relation stats: total(10046), max(10349000) in 139s
```

This is a stat from osm2pgsql that shipped with Nominatim for comparison
```
Processing: Node(2000k 125.0k/s) Way(62k 3.44k/s) Relation(0 0.00/s)
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1014#issuecomment-562836469
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20191207/0372e434/attachment-0001.html>