[Tile-serving] [osm2pgsql] large postgres shared_buffers slows down import (#163)

[Hannes Janetzek]

Hi, I've started a planet import on a quite beefy server at university which has 256GB of ram - I read some recommendations for shared_buffers to be up to 1/4 of available memory - so I used 64GB. 

Though the relation read speed was ~6/sec which (not only) compared to the other benchmarks seemed too slow.

Then I've attached 'perf' to the various processes and found that all time of postgres was spent within  FlushRelationBuffers (not the osm relations :) at some eax/mov. Looking into the source[1] I guessed that it could be the loop for NBuffers. Looking a bit further NBuffers  seemed to depend on shared_buffers - In short, reducing shared_buffers to 512MB increased the relation read speed to 170/sec :)

Still I wonder why reading in relations triggers a end_heap_write (probably for each relation)[2]

Maybe you could add to the README the hint to not increase shared_buffers over 1GB :)

[1]http://www.comp.nus.edu.sg/~cs3223/doxygen-postgres9.3.2/html/d1/d20/bufmgr-original_8c_source.html#l02370
[2]http://www.comp.nus.edu.sg/~cs3223/doxygen-postgres9.3.2/html/d1/d20/bufmgr-original_8c.html#ac46973f353ef3fab852b5bd90ed87dde


---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/163
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20140625/1df6407a/attachment.html>