[OSM-dev] osm2pgsql memory usage

Jon Burgess jburgess777 at gmail.com
Mon Feb 7 22:10:17 GMT 2011


On Mon, 2011-02-07 at 20:00 +0100, Frederik Ramm wrote:
> Hi,
> 
>     when you run osm2pgsql in slim mode with a large cache, then could 
> it not release the cache memory once it reaches the "Sorting data and 
> creating indexes" stage? It is this phase where postgresql tends to 
> consume a lot of memory for index creation, and the specific settings on 
> the machine I'm running it on at the moment have led to all of 
> osm2pgsql's 10 GB being swapped out - which, if the memory is really 
> unused, isn't a big deal but I wonder if the memory could not have been 
> freed anyway?

I think what you are describing is an issue I am aware of and even have
a patch which I use myself (attached).

The memory allocations are being freed correctly but the overall 'sbrk'
memory can not be released due to fragmentation. With the attached patch
the block allocator makes larger malloc requests which glibc fulfils via
an mmap() based scheme which avoids this fragmentation (see
MMAP_THRESHOLD in the malloc man page).

The reason that I have not applied this to the main tree is that it
triggers more out-of-memory errors for people that are importing small
extracts. A better long term solution would be some way of automatically
detecting whether the data being imported seems to be dense (i.e. a
whole planet) vs sparse (a country extract) and tweaking some of the
internal parameters to optimize for these two situations.

   Jon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: osm2pgsql-memory.patch
Type: text/x-patch
Size: 652 bytes
Desc: not available
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20110207/44451378/attachment.bin>


More information about the dev mailing list