<div dir="ltr">Hello Stephan,<div><br></div><div>Darafei from PostGIS here. Correct spot to report this is <a href="http://trac.osgeo.org/postgis" target="_blank">http://trac.osgeo.org/postgis</a>.</div><div>Please note your proj is 5 (from 2018) and that's 3 years old, there was an API change to accomodate newer proj versions.</div><div><br></div><div>What we'll need is a way to reproduce the issue. Just "tirex" is not enough, a docker-compose with all the scripts running or isolated query that causes the leak will help. At the very least - enable a full log of queries with auto_explain and zero threshold and attach the postgresql.log with all these, so we have something to look into.</div><div><br></div><div>Thanks.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Mar 27, 2021 at 4:52 PM Stephan Knauss <<a href="mailto:osm@stephans-server.de" target="_blank">osm@stephans-server.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have recently updated my rendering stack to latest software releases.<br>
Now I observe a memory leak of about 2GB of available memory per hour.<br>
<br>
Memory seems to stay allocated with PostgreSQL processes. These have <br>
open connections with the tirex backends doing the rendering.<br>
There is no idle time of the connections as I have a full rendering queue.<br>
<br>
Is this something others face as well? I do not remember such memory <br>
leaks from my previous setup. Only versions of the software stack in <br>
place changed.<br>
<br>
Stopping the rendering queue and restarting tirex-backends will release <br>
the memory.<br>
I could do this as a work-around, but I am suspecting that this is no <br>
normal behavior.<br>
<br>
Are there ways to debug what causes Postgresql to keep the memory?<br>
Did I miss some configuration option?<br>
<br>
Posgres/Postgis is from their latest docker image postgis/postgis (hash <br>
74a85c5bd6ac)<br>
<br>
PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, <br>
compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit<br>
<br>
POSTGIS=""3.1.1 aaf4c79"" [EXTENSION] PGSQL=""130"" <br>
GEOS=""3.7.1-CAPI-1.11.1 27a5e771"" PROJ=""Rel. 5.2.0, September 15th, <br>
2018"" LIBXML=""2.9.4"" LIBJSON=""0.12.1"" LIBPROTOBUF=""1.3.1"" <br>
WAGYU=""0.5.0 (Internal)""<br>
<br>
<br>
Relevant syslog of OOM:<br>
<br>
postgres invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), <br>
order=0, oom_score_adj=0<br>
CPU: 9 PID: 3322620 Comm: postgres Not tainted 5.4.0-67-generic #75-Ubuntu<br>
Hardware name: Gigabyte Technology Co., Ltd. B360 HD3P-LM/B360HD3PLM-CF, <br>
BIOS F7 HZ 07/24/2020<br>
Call Trace:<br>
dump_stack+0x6d/0x8b<br>
dump_header+0x4f/0x1eb<br>
oom_kill_process.cold+0xb/0x10<br>
out_of_memory.part.0+0x1df/0x3d0<br>
out_of_memory+0x6d/0xd0<br>
__alloc_pages_slowpath+0xd5e/0xe50<br>
__alloc_pages_nodemask+0x2d0/0x320<br>
alloc_pages_current+0x87/0xe0<br>
__page_cache_alloc+0x72/0x90<br>
pagecache_get_page+0xbf/0x300<br>
filemap_fault+0x6b2/0xa50<br>
? unlock_page_memcg+0x12/0x20<br>
? page_add_file_rmap+0xff/0x1a0<br>
? xas_load+0xd/0x80<br>
? xas_find+0x17f/0x1c0<br>
? filemap_map_pages+0x24c/0x380<br>
ext4_filemap_fault+0x32/0x50<br>
__do_fault+0x3c/0x130<br>
do_fault+0x24b/0x640<br>
? __switch_to_asm+0x34/0x70<br>
__handle_mm_fault+0x4c5/0x7a0<br>
handle_mm_fault+0xca/0x200<br>
do_user_addr_fault+0x1f9/0x450<br>
__do_page_fault+0x58/0x90<br>
do_page_fault+0x2c/0xe0<br>
page_fault+0x34/0x40<br>
RIP: 0033:0x558ddf0beded<br>
Code: Bad RIP value.<br>
RSP: 002b:00007ffe5214a020 EFLAGS: 00010202<br>
RAX: 00007fea26b16b28 RBX: 0000000000000028 RCX: 00007fea26b16b68<br>
RDX: 0000000000000028 RSI: 0000000000000000 RDI: 00007fea26b16b28<br>
RBP: 0000000000000010 R08: 00007fea26b16b28 R09: 0000000000000019<br>
R10: 0000000000000001 R11: 0000000000000001 R12: 00000000ffffffff<br>
R13: 00007fea26af76d8 R14: 00007fea26af7728 R15: 0000000000000000<br>
Mem-Info:<br>
active_anon:29797121 inactive_anon:2721653 isolated_anon:32<br>
active_file:323 inactive_file:83 isolated_file:0<br>
unevictable:16 dirty:14 writeback:0 unstable:0<br>
slab_reclaimable:85925 slab_unreclaimable:106003<br>
mapped:1108591 shmem:14943591 pagetables:69567 bounce:0<br>
free:148637 free_pcp:1619 free_cma:0<br>
Node 0 active_anon:119188484kB inactive_anon:10886612kB <br>
active_file:1292kB inactive_file:332kB unevictable:64kB <br>
isolated(anon):128kB isolated(file):0kB mapped:4434364kB dirty:56kB <br>
writeback:0kB shmem:59774364kB shmem_thp: 0kB shmem_pmdmapped: 0kB <br>
anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no<br>
Node 0 DMA free:15904kB min:8kB low:20kB high:32kB active_anon:0kB <br>
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB <br>
writepending:0kB present:15988kB managed:15904kB mlocked:0kB <br>
kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB <br>
free_cma:0kB<br>
lowmem_reserve[]: 0 809 128670 128670 128670<br>
Node 0 DMA32 free:511716kB min:424kB low:1252kB high:2080kB <br>
active_anon:328544kB inactive_anon:8540kB active_file:204kB <br>
inactive_file:0kB unevictable:0kB writepending:0kB present:947448kB <br>
managed:881912kB mlocked:0kB kernel_stack:0kB pagetables:416kB <br>
bounce:0kB free_pcp:1412kB local_pcp:288kB free_cma:0kB<br>
lowmem_reserve[]: 0 0 127860 127860 127860<br>
Node 0 Normal free:66928kB min:67148kB low:198076kB high:329004kB <br>
active_anon:118859812kB inactive_anon:10877988kB active_file:1812kB <br>
inactive_file:1480kB unevictable:64kB writepending:56kB <br>
present:133160960kB managed:130937320kB mlocked:64kB <br>
kernel_stack:18336kB pagetables:277852kB bounce:0kB free_pcp:5064kB <br>
local_pcp:384kB free_cma:0kB<br>
lowmem_reserve[]: 0 0 0 0 0<br>
Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB <br>
(U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB<br>
Node 0 DMA32: 1388*4kB (UME) 1274*8kB (UME) 1772*16kB (UE) 1564*32kB <br>
(UME) 1082*64kB (UE) 514*128kB (UME) 160*256kB (UE) 46*512kB (UME) <br>
23*1024kB (UE) 11*2048kB (UME) 42*4096kB (UME) = 511808kB<br>
Node 0 Normal: 715*4kB (UEH) 108*8kB (UEH) 1400*16kB (UMEH) 1156*32kB <br>
(UMEH) 58*64kB (UMEH) 7*128kB (UMEH) 1*256kB (U) 0*512kB 0*1024kB <br>
0*2048kB 0*4096kB = 67980kB<br>
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 <br>
hugepages_size=1048576kB<br>
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 <br>
hugepages_size=2048kB<br>
14982418 total pagecache pages<br>
38157 pages in swap cache<br>
Swap cache stats: add 14086008, delete 14047525, find 102734363/105748939<br>
Free swap = 0kB<br>
Total swap = 4189180kB<br>
33531099 pages RAM<br>
0 pages HighMem/MovableOnly<br>
572315 pages reserved<br>
0 pages cma reserved<br>
0 pages hwpoisoned<br>
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=fae7f79d710f4449fd87c58f38eb164a470e3f837b33630c6c10a9fbca10a82b,mems_allowed=0,global_oom,task_memcg=/docker/e88b65d20ef39588c6bf9c00e7aa2946f134a61a6195c210f7081d7ed4d9a5fa,task=postgres,pid=2453393,uid=999<br>
Out of memory: Killed process 2453393 (postgres) total-vm:10625568kB, <br>
anon-rss:6768188kB, file-rss:4kB, shmem-rss:3592772kB, UID:999 <br>
pgtables:20756kB oom_score_adj:0<br>
oom_reaper: reaped process 2453393 (postgres), now anon-rss:0kB, <br>
file-rss:0kB, shmem-rss:3592772kB<br>
<br>
<br>
<br>
<br>
<br>
Stephan<br>
<br>
_______________________________________________<br>
dev mailing list<br>
<a href="mailto:dev@openstreetmap.org" target="_blank">dev@openstreetmap.org</a><br>
<a href="https://lists.openstreetmap.org/listinfo/dev" rel="noreferrer" target="_blank">https://lists.openstreetmap.org/listinfo/dev</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Darafei "Komяpa" Praliaskouski<br>OSM BY Team - <a href="http://openstreetmap.by/" target="_blank">http://openstreetmap.by/</a><br></div></div>