[Tile-serving] [osm2pgsql] Explain why fork() failed (#47)

[apmon]

Yes, I haven't seen any other reason for the fork to fail than "out of memory" errors.

The issue is that if you run with e.g. -c 16000 --number-process 8, at the time of fork, the osm2pgsql process will have 16GB of memory allocated (and used if it is a full planet import). If you know forking the process 7 times, you are asking the OS for 8 * 16GB (128GB),  which is likely way more than what you have installed in your server.

As osm2pgsql never writes to those 16GB of memory and all modern OSs have copy-on-write for fork(), it never actually uses more than the ~16GB of ram. But your OS doesn't know that at the time of fork and so might reject it depending on your kernels overcommit behaviour.


The correct thing to do would be to use shared memory, so that the OS does know that osm2pgsql never uses more than the cache once and thus not run into potential problems at the time of fork().

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/pull/47#issuecomment-19861718
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20130622/94bbc346/attachment.html>