[Tile-serving] [openstreetmap/osm2pgsql] Possible performance improvement (#1532)

[Stephan]

"Working set" indicates that these are actually pages mapped to physical. So this is RAM in use by the process and not available to others unless paged out to disk (or compressed, Windows does similar to ZRAM).
As Jochen pointed out, this does not necessarily mean that the application code is not releasing the memory, it could also be the behavior of the allocator in use.

See this for a nice explanation on how for example the glibc allocator works and where I had analyzed such issues before: https://sourceware.org/glibc/wiki/MallocInternals

Potentially you are facing such an issue. Further investigation here would certainly require some additional time and effort. It would specifically get more complicated in case the effect is limited to Windows, as to my knowledge the main contributors work with Linux.

To move here forward I recommend that you create a reproduction case. Potentially pausing osm2pgsql execution at the right point  (after "Reading input files done"). If this again confirms that the allocator does not return memory to the OS, despite huge amounts of memory are released by application code then further steps can be considered. Allocators can be tuned or replaced by other allocators like for example tcmalloc. Would also require some investigation what allocator implementation is actually used in your windows build.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/openstreetmap/osm2pgsql/issues/1532#issuecomment-876977594
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/tile-serving/attachments/20210709/776883df/attachment.htm>