[Tile-serving] [openstreetmap/osm2pgsql] possible memoryleak with 1.5.0 (#1539)

[ಠ_ಠ]

it seems there is some sort of memory leak with the current release taken from the debian experimental repository.

```
$ osm2pgsql --log-level=debug --hstore-all --slim --drop --cache=16000 -G --number-processes=4 --flat-nodes=tmp -H 192.168.2.103 -U osm -d osm planet-latest.osm.pbf
2021-07-16 21:05:06  osm2pgsql version 1.5.0
2021-07-16 21:05:06  [0] Database version: 13.3 (Debian 13.3-1)
2021-07-16 21:05:06  [0] PostGIS version: 3.1
2021-07-16 21:05:06  [0] Reading file: planet-latest.osm.pbf
2021-07-16 21:05:06  [0] Started pool with 4 threads.
2021-07-16 21:05:06  [0] Loading persistent node cache from 'tmp'.
2021-07-16 21:05:06  [0] Mid: pgsql, cache=16000
2021-07-16 21:05:06  [0] Setting up table 'planet_osm_nodes'
2021-07-16 21:05:06  [0] Setting up table 'planet_osm_ways'
2021-07-16 21:05:07  [0] Setting up table 'planet_osm_rels'
2021-07-16 21:05:08  [0] Using projection SRS 3857 (Spherical Mercator)
2021-07-16 21:05:08  [0] Using built-in tag transformations
2021-07-16 21:05:08  [0] Setting up table 'planet_osm_point'
2021-07-16 21:05:10  [0] Setting up table 'planet_osm_line'
2021-07-16 21:05:11  [0] Setting up table 'planet_osm_polygon'
2021-07-16 21:05:11  [0] Setting up table 'planet_osm_roads'
Processing: Node(2111400k 5571.0k/s) Way(0k 0.00k/s) Relation(0 0.0/s)[1]    181885 killed
```

the system has 32GB of memory and i was able to observe the crash with htop.
it honored the memorylimit of the cache paramter until it reached that last node seen in the output, it stopped updating and processing the file but increased until system memory was full and it was killed by the OOM.

i restarted the process with osm2pgsql 1.4.1 and it seems to run just fine beyond this point.

the planetfile ( from https://planet.osm.org/pbf/planet-latest.osm.pbf ) in question is this:

```
$ osmium fileinfo -e planet-latest.osm.pbf
File:
  Name: planet-latest.osm.pbf
  Format: PBF
  Compression: none
  Size: 62962428417
Header:
  Bounding boxes:
    (-180,-90,180,90)
  With history: no
  Options:
    generator=planet-dump-ng 1.2.0
    osmosis_replication_timestamp=2021-06-27T23:59:58Z
    pbf_dense_nodes=true
    pbf_optional_feature_0=Has_Metadata
    pbf_optional_feature_1=Sort.Type_then_ID
    sorting=Type_then_ID
    timestamp=2021-06-27T23:59:58Z
[======================================================================] 100%
Data:
  Bounding box: (-180,-90,180,90)
  Timestamps:
    First: 2005-05-03T13:27:18Z
    Last: 2021-06-27T23:59:58Z
  Objects ordered (by type and id): yes
  Multiple versions of same object: no
  CRC32: not calculated (use --crc/-c to enable)
  Number of changesets: 0
  Number of nodes: 7041839672
  Number of ways: 782228827
  Number of relations: 9054640
  Smallest changeset ID: 0
  Smallest node ID: 1
  Smallest way ID: 37
  Smallest relation ID: 11
  Largest changeset ID: 0
  Largest node ID: 8870746351
  Largest way ID: 958713294
  Largest relation ID: 12893861
  Number of buffers: 10680839 (avg 733 objects per buffer)
  Sum of buffer sizes: 698105454040 (698.105 GB)
  Sum of buffer capacities: 701029613568 (701.029 GB, 100% full)
Metadata:
  All objects have following metadata attributes: version+timestamp+changeset
  Some objects have following metadata attributes: all

```




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1539
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210716/58c60ca6/attachment.htm>