[Tile-serving] [openstreetmap/osm2pgsql] Potential memory leak when running without flatnode option (#931)

[jbone80]

I am new to this osm2pgsql, work required I learn it.  Bear with the first part, but it was part of the process I was using to have all the memory consumed.  I was running into this error:

`F:\osm2pgsql-bin>osm2pgsql C:\planet-190415.osm.pbf -v -c -d osm -U postgres -W --hstore
osm2pgsql version 0.96.0 (64 bit id space)

Password:
Allocating memory for sparse node cache
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=1
Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: C:\planet-190415.osm.pbf
Using PBF parser.
Processing: Node(41200k 5885.7k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Node cache size is too small to fit all nodes. Please increase cache size
Error occurred, cleaning up`

So I started running it with the -C flag set to 40000 and this is what I get when it is running:

![Memory_consuming](https://user-images.githubusercontent.com/40067691/61464160-0becb800-a933-11e9-8cad-8715796426dd.JPG)

Windows is telling me osm2pgsql is consuming all of the memory, but even with the -C flag set to 40000 I am still getting this:

`F:\osm2pgsql-bin>osm2pgsql C:\planet-190415.osm.pbf -v -c -d osm -U postgres -W -C 40000 --hstore
osm2pgsql version 0.96.0 (64 bit id space)

Password:
Allocating memory for sparse node cache
Node-cache: cache=40000MB, maxblocks=640000*65536, allocation method=1
Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: C:\planet-190415.osm.pbf
Using PBF parser.
Processing: Node(2619980k 5446.9k/s) Way(0k 0.00k/s) Relation(0 0.00/s)
Node cache size is too small to fit all nodes. Please increase cache size
Error occurred, cleaning up`

For some reason today it isn't dying, it killed my inject yesterday 3 times.  Yesterday it would kill the process before it finished loading all the nodes, before it transitions to loading the "Way" portion of it. 
 When I add the -F and -s flag it works just fine.   All I have to do is add the -F flag and it works, I haven't specified a node file for it to use at all.  However, I have it stored on a drive that is 400GB and it consumes every last bit of that during the injection and dies because it runs out of space.  I don't know how much space I should give it. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/931#issuecomment-512834944
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20190718/7756cd72/attachment.html>