OOM killer terminates cbq-engine


I’m very new in Couchbase, so sorry for possible errors :).

There is a Couchbase Community Edition 5.0.1-5003-1 from Debian, with 20G RAM. The operation system runs on a Xen virtual machine - it’s a Debian 9.

Sometimes the OOM kills the cbq-engine. The course starts like this:

beam.smp invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
beam.smp cpuset=/ mems_allowed=0
CPU: 3 PID: 1048 Comm: beam.smp Not tainted 4.9.0-6-amd64 #1 Debian 4.9.82-1+deb9u3
active_anon:4649138 inactive_anon:357654 isolated_anon:0
 active_file:112 inactive_file:145 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 slab_reclaimable:4926 slab_unreclaimable:10268
 mapped:1386 shmem:51446 pagetables:16139 bounce:0
 free:24658 free_pcp:190 free_cma:0
51973 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 4194300kB
Total swap = 4194300kB
5242783 pages RAM
0 pages HighMem/MovableOnly
105740 pages reserved
0 pages hwpoisoned
Out of memory: Kill process 13773 (cbq-engine) score 319 or sacrifice child
Killed process 13773 (cbq-engine) total-vm:8592000kB, anon-rss:7894568kB, file-rss:0kB, shmem-rss:0kB

There is a small script, which runs in every minute, and checks the memory allocation. If there are any process, which uses much memory, starts to make the “snapshot” about the output of top command. Here is the last:

Wed Jan 22 17:50:02 CET 2020
top - 17:50:02 up 224 days,  5:22,  0 users,  load average: 25.62, 7.66, 3.17
Tasks: 217 total,   2 running, 214 sleeping,   0 stopped,   1 zombie
%Cpu(s):  5.1 us,  2.5 sy,  0.0 ni, 92.1 id,  0.1 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:  20548172 total, 19533836 used,  1014336 free,     4052 buffers
KiB Swap:  4194300 total,        0 used,  4194300 free.   302696 cached Mem

11927 couchba+  20   0 7754484 4.848g      0 S   6.1 24.7 220:55.94 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
24803 wildfly   20   0 4674524 2.650g      0 S   6.1 13.5 505:29.54 java -D[Standalone] -server  Duser.timezone=Europe/Budapest -Dfile.encoding=utf-8 -Xms2048m -Xmx2048M -XX:MetaspaceSize=512M  XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman  Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/if3/wildfly/standalone/log/server.log  Dlogging.configuration=file:/opt/if3/wildfly/standalone/configuration/logging.properties -jar /opt/if3/wildfly/jboss-m+
     total       used       free     shared    buffers     cached
Mem:         20066      19038       1028        200          3        295
-/+ buffers/cache:      18738       1327
Swap:         4095          0       4095

As you can see, the 19G of 20G total are used - but there isn’t any SWAP memory used. I know, this is an another issue, but if anybody has an idea about what can cause this heavy memory usage, please help me.

If you have any question, just let me know.



cbq-engine is victim. To solve this you need to figure out what queries you run and create right index using monitoring tools.

Optimize N1QL Performance Using Request Profiling, Query Performance Monitoring Made Easy With Couchbase N1QL