[OSM-dev] Memory-Leak in osmium

[Jochen Topf]

On Mon, Nov 21, 2011 at 12:07:42PM +0100, Peter Körner wrote:
> I found a memory-leak in osmium-master. I compiled the
> osmium_progress example and fed germany.osh.pbf <http://ftp5.gwdg.de/pub/misc/openstreetmap/osm-full-history-extracts/110919/pbf/europe/germany.osh.pbf>
> into it.
> 
> During the nodes processing the program takes 70 MB of RAM. When it
> starts to process Ways, the required memory raises constantly up to
> 300 MB.

I don't think that there is a memory leak. Valgrind tells me that all is ok.

The behaviour you see comes from the PBF reader and especially the Google
Protobuf stuff. Protobuf does not free memory but keeps it around until the
next time it is needed. So every time you hit a PBF block with "more" data in
it than all the other blocks before (for instance more nodes per way), PBF will
allocated more memory and not de-allocate it. So for a while you'll see the
process slowly using more and more memory until it hits the peak.

Only google::protobuf::ShutdownProtobufLibrary(); will release all the memory.
Thats done at the end of the program. (Could be done at the end of the read,
but for most programs thats the same as the end of the program anyway.) So
there is no leak, all memory is accounted for.

Jochen
-- 
Jochen Topf  jochen at remote.org  http://www.remote.org/jochen/  +49-721-388298