[Tile-serving] [openstreetmap/osm2pgsql] possible memoryleak with 1.5.0 (#1539)

[ಠ_ಠ]

with cache set to 0 as you suggested it runs just fine...
```
osm2pgsql --log-level=debug --hstore-all --slim --drop --cache=0 -G --number-processes=4 --flat-nodes=tmp -H 192.168.2.103 -U osm -d osm2 planet-latest.osm.pbf
2021-07-18 10:50:56  osm2pgsql version 1.5.0
2021-07-18 10:50:57  [0] Database version: 13.3 (Debian 13.3-1)
2021-07-18 10:50:57  [0] PostGIS version: 3.1
2021-07-18 10:50:57  [0] Reading file: planet-latest.osm.pbf
2021-07-18 10:50:57  [0] Started pool with 4 threads.
2021-07-18 10:50:57  [0] Loading persistent node cache from 'tmp'.
2021-07-18 10:50:57  [0] Mid: pgsql, cache=0
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_nodes'
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_ways'
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_rels'
2021-07-18 10:50:57  [0] Using projection SRS 3857 (Spherical Mercator)
2021-07-18 10:50:57  [0] Using built-in tag transformations
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_point'
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_line'
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_polygon'
2021-07-18 10:50:57  [0] Setting up table 'planet_osm_roads'
Processing: Node(6701520k 6131.3k/s) Way(0k 0.00k/s) Relation(0 0.0/s)^C
```

with set to 16000 it fails:
```
osm2pgsql --log-level=debug --hstore-all --slim --drop --cache=16000 -G --number-processes=4 --flat-nodes=tmp -H 192.168.2.103 -U osm -d osm2 planet-latest.osm.pbf
2021-07-18 11:09:15  osm2pgsql version 1.5.0
2021-07-18 11:09:15  [0] Database version: 13.3 (Debian 13.3-1)
2021-07-18 11:09:15  [0] PostGIS version: 3.1
2021-07-18 11:09:15  [0] Reading file: planet-latest.osm.pbf
2021-07-18 11:09:15  [0] Started pool with 4 threads.
2021-07-18 11:09:15  [0] Loading persistent node cache from 'tmp'.
2021-07-18 11:09:15  [0] Mid: pgsql, cache=16000
2021-07-18 11:09:15  [0] Setting up table 'planet_osm_nodes'
2021-07-18 11:09:15  [0] Setting up table 'planet_osm_ways'
2021-07-18 11:09:20  [0] Setting up table 'planet_osm_rels'
2021-07-18 11:09:21  [0] Using projection SRS 3857 (Spherical Mercator)
2021-07-18 11:09:21  [0] Using built-in tag transformations
2021-07-18 11:09:21  [0] Setting up table 'planet_osm_point'
2021-07-18 11:09:22  [0] Setting up table 'planet_osm_line'
2021-07-18 11:09:22  [0] Setting up table 'planet_osm_polygon'
2021-07-18 11:09:22  [0] Setting up table 'planet_osm_roads'
Processing: Node(2111760k 5373.4k/s) Way(0k 0.00k/s) Relation(0 0.0/s)[1]    3622294 killed osm2pgsql --log-level=debug --hstore-all --slim --drop
```

that sudden spike in memory consumptions halts the progress for about 4-5 seconds until it gets killed.

with the cache set to its default ( 800 ) it does not fail.

```
11:20:17 » easteregg at t480 ~/osm-import $ osm2pgsql --log-level=debug --hstore-all --slim --drop --cache=800 -G --number-processes=4 --flat-nodes=tmp -H 192.168.2.103 -U osm -d osm2 planet-latest.osm.pbf
2021-07-18 11:20:27  osm2pgsql version 1.5.0
2021-07-18 11:20:27  [0] Database version: 13.3 (Debian 13.3-1)
2021-07-18 11:20:27  [0] PostGIS version: 3.1
2021-07-18 11:20:27  [0] Reading file: planet-latest.osm.pbf
2021-07-18 11:20:27  [0] Started pool with 4 threads.
2021-07-18 11:20:27  [0] Loading persistent node cache from 'tmp'.
2021-07-18 11:20:27  [0] Mid: pgsql, cache=800
2021-07-18 11:20:27  [0] Setting up table 'planet_osm_nodes'
2021-07-18 11:20:27  [0] Setting up table 'planet_osm_ways'
2021-07-18 11:20:28  [0] Setting up table 'planet_osm_rels'
2021-07-18 11:20:28  [0] Using projection SRS 3857 (Spherical Mercator)
2021-07-18 11:20:28  [0] Using built-in tag transformations
2021-07-18 11:20:28  [0] Setting up table 'planet_osm_point'
2021-07-18 11:20:28  [0] Setting up table 'planet_osm_line'
2021-07-18 11:20:28  [0] Setting up table 'planet_osm_polygon'
2021-07-18 11:20:28  [0] Setting up table 'planet_osm_roads'
Processing: Node(3302180k 5185.1k/s) Way(0k 0.00k/s) Relation(0 0.0/s)^C
```


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1539#issuecomment-882028383
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210718/d2909d75/attachment.htm>