[Tile-serving] [osm2pgsql] OOM killed after processing phase (#351)

[Paul Norman]

> The most likely candidate for the jump in allocations above however is the pending queue. All pending ways are initially put in the queue with each entry taking up 16 byte (an osmid and a pointer). I haven't tested with a full planet but for extracts about 60% of the ways need to be reprocessed, so for the planet about 160M ways. That leaves the queue with a total of 2.5G of memory. Assuming that queue behaves like a vector which doubles it's size during each reallocation, the queue might easily take up 5G of memory (make that 7.5GB shortly during reallocation when the vector is expanded).

So in processing we're building them up into a different data structure, then in pending, copying from that to a less space-efficient queue?

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/351#issuecomment-106573478
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150528/8d12f954/attachment.html>