[Tile-serving] [openstreetmap/osm2pgsql] Troubleshooting of the planet import (#785)

[hz]

When I do this on an i2-4xlarge EC2 instance (16proc, 122gb RAM) I get 'Killed' while processing Ways out of the PBF file.

` osm2pgsql -r pbf -c -S magrathea.style -k -C 43000 -I --unlogged --cache-strategy dense --flat-nodes flat.nodes --number-processes 16 -H 127.0.0.1 --prefix planet_osm -U gisuser -d osmplanet planet-latest.osm.pbf`

-C 30000 failed with out of node cache memory after processing 3000000k nodes.  Since there are 4146299k nodes i went with 43000 to be on the safe side.

> osm2pgsql version 0.95.0-dev (64 bit id space)
> 
> Using built-in tag processing pipeline
> Using projection SRS 3857 (Spherical Mercator)
> Setting up table: planet_osm_point
> Setting up table: planet_osm_line
> Setting up table: planet_osm_polygon
> Setting up table: planet_osm_roads
> Allocating memory for dense node cache
> Allocating dense node cache in one big chunk
> Node-cache: cache=43000MB, maxblocks=688000*65536, allocation method=2
> 
> Reading in file: planet-latest.osm.pbf
> Using PBF parser.
> Processing: Node(4146299k 3960.2k/s) Way(119130k 46.46k/s) Relation(0 0.00/s)Killed

Built from source from master today.  I had a similar issue with importing just North America a few months ago and the process grew to about 90GB of RAM (-C much smaller number though).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/785#issuecomment-340587473
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20171030/4a6fff10/attachment.html>