[Tile-serving] [openstreetmap/osm2pgsql] Settings for importing in 2019 for the whole planet (#962)

[Georg Bachmann]

I am currently having huge problems importing the whole planet. We are currently already waiting more than 1.5 days and it's still not done.
 And our machine is quite fast. We have 250GB RAM and 32 Cores. And all is running on SSDs...

I've read that the `--slim` should be used for the whole planet. Without this param we went out of memory quite fast.

So this is how our import statement looks currently:
```
osm2pgsql -S my.style -c -H xxx-d x -U xxx --slim --drop --cache 100000 --number-processes 8 planet.osm.pbf
```

The problem is that currently there seems to be a lot of postgres threads going on. `CREATE TABLE`, `COMMIT`, `PREPARE insert_node/way/rel`... those kind of queries.

But those are stuck waiting for IO.
We did define that osm2psql can use about 100GB of ram, but it's currently "only" using about 50GB?!?

Here's an extract of our postgres-config:
```
shared_buffers = 4GB
work_mem = 2MB 
maintenance_work_mem = 2GB
dynamic_shared_memory_type = posix
effective_io_concurrency = 200
max_worker_processes = 30
max_parallel_workers_per_gather = 15
wal_buffers = 16MB
min_wal_size = 4GB
max_wal_size = 8GB
checkpoint_completion_target = 0.9
effective_cache_size = 12GB
default_statistics_target = 500
```

Is there an obvious mistake we are making? We are happy about every hint.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/962
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20191017/f00652ba/attachment.html>