[Tile-serving] [osm2pgsql] OOM killed after processing phase (#351)

[Sarah Hoffmann]

There is quite a bit of memory allocation going on before processing of pending. The memory cache for the flatnode store alone allocates about 650MB (80MB per thread).

The most likely candidate for the jump in allocations above however is the pending queue. All pending ways are initially put in the queue with each entry taking up 16 byte (an osmid and a pointer). I haven't tested with a full planet but for extracts about 60% of the ways need to be reprocessed, so for the planet about 160M ways. That leaves the queue with a total of 2.5G of memory. Assuming that queue behaves like a vector which doubles it's size during each reallocation, the queue might easily take up 5G of memory (make that 7.5GB shortly during reallocation when the vector is expanded).

It should be possible to avoid the allocation of cache memory for the flatnode store when there is enough ram cache. The pending queue issue is more difficult to fix. A size-bound queue where middle acts as a producer is probably the most reasonable solution but not exactly easy to implement. (And just to mention it, there is of course also the solution to drop support for old-style multipolygons. Then we could just get rid of all this processing of pending ways and get a huge performance improvement on the side.)

@pnorman Any idea how to confirm the theory with the pending queue on a planet import? 

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/351#issuecomment-106569548
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150528/1af1bfa0/attachment.html>