[Tile-serving] [osm2pgsql] OOM killed after processing phase (#351)

[Paul Norman]

On a machine with 32GB RAM, osm2pgsql 0.87.3, command line `~/osm2pgsql/osm2pgsql -d gis --cache 26000 --flat-nodes nodes.bin --slim --drop --
number-processes 8 -S ~/openstreetmap-carto/openstreetmap-carto.style planet-150202.osm.pbf`

```
osm2pgsql SVN version 0.87.3 (64bit id space)

Using built-in tag processing pipeline
Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=26000MB, maxblocks=3328000*8192, allocation method=11
Mid: loading persistent node cache from nodes.bin
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0
Mid: pgsql, scale=100 cache=26000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: planet-150202.osm.pbf
Processing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2630170 269.21/s)
Standard exception processing relation id=3890452: TopologyException: side location conflict at 1038877.48 5779728.6200000001
Processing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2841950 273.71/s)
Standard exception processing relation id=4136486: TopologyException: side location conflict at 2924456.4100000001 9012036.7799999993
Processing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(2881850 275.04/s)
Standard exception processing relation id=4180656: TopologyException: side location conflict at 272703.48999999999 5093940.6600000001
Processing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(3100780 281.58/s)
Standard exception processing relation id=4542207: TopologyException: side location conflict at 15042893.51 4046892.75
Processing: Node(2715748k 2596.3k/s) Way(272021k 51.88k/s) Relation(3109640 282.05/s)  parse time: 17314s
Node stats: total(2715748725), max(3326469484) in 1046s
Way stats: total(272021914), max(326028236) in 5243s
Relation stats: total(3109645), max(4551629) in 11025s
Maximum node in persistent node cache: 3327131647
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Command terminated by signal 9
```

Watching `status` in /proc/PID tells me that about 24500 cache would have been sufficient.

There are about 8GB free as it finishes processing

![image](https://cloud.githubusercontent.com/assets/1190866/7776708/de58f4b2-0071-11e5-8566-602376c3f0f1.png)

I haven't yet looked through the code to see what could be using large amounts of memory. `vm.overcommit_memory` is set to 1, so an allocation alone won't cause a problem, only use. `vm.overcommit_ratio` is 50, but I don't believe that matters with overcommit_memory set to 1.

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/351
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150522/78a841a3/attachment.html>