[Tile-serving] [osm2pgsql] OOM killed after processing phase (#351)

[Paul Norman]

> > So in processing we're building them up into a different data structure, then in pending, copying from that to a less space-efficient queue?
> Jup, that's what I read from in the code.

Two thoughts then

1. Read directly from the initial data structure instead of the queue. That would mean making the initial data structure thread-safe and efficient at reading from it in parallel. Or having each thread only read ways that have way_id mod num-threads = thread number (or similar logic working on groups of 1k ways)

2. Copying into the queue in chunks, copying in a new chunk whenever it gets empty.

---
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/351#issuecomment-106576752
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20150528/9fc41fb2/attachment.html>