[2497237.628214] Task in /kubepods/burstable/podf3156ead-9d31-4d00-bf89-5e781ab2dff2/9cefe4385170021c2abb27b2b9415651b5eb829fe15fb7f3c077ba31554c8c6c killed as a result of limit of /kubepods/burstable/podf3156ead-9d31-4d00-bf89-5e781ab2dff2/9cefe4385170021c2abb27b2b9415651b5eb829fe15fb7f3c077ba31554c8c6c
[2497237.654886] memory: usage 64487424kB, limit 64487424kB, failcnt 72410358
[2497237.661770] memory+swap: usage 64487424kB, limit 64487424kB, failcnt 22015
[2497237.668828] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
[2497237.675018] Memory cgroup stats for /kubepods/burstable/podf3156ead-9d31-4d00-bf89-5e781ab2dff2/9cefe4385170021c2abb27b2b9415651b5eb829fe15fb7f3c077ba31554c8c6c: cache:8588472KB rss:55898952KB rss_huge:655360KB mapped_file:48KB swap:0KB inactive_anon:0KB active_anon:55898956KB inactive_file:4296904KB active_file:4288724KB unevictable:0KB
[2497237.716261] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[2497237.724789] [123005] 0 123005 3000 437 12 0 837 couchentry.sh
[2497237.733767] [123078] 0 123078 1096 123 7 0 837 runsvdir
[2497237.742300] [123100] 0 123100 1058 88 7 0 837 runsv
[2497237.750578] [123156] 0 123156 2907 113 11 0 837 epmd
[2497237.758770] [141122] 0 141122 28245 384 56 0 -1000 sshd
[2497237.766953] [145597] 0 145597 3466348 29507 464 0 837 beam.smp
[2497237.775481] [147486] 0 147486 1089 130 7 0 837 erl_child_setup
[2497237.784619] [149067] 0 149067 2463 555 9 0 837 gosecrets
[2497237.793234] [149117] 0 149117 4224154 720732 2144 0 837 beam.smp
[2497237.801759] [149195] 0 149195 1089 131 8 0 837 erl_child_setup
[2497237.810901] [150757] 0 150757 2924 342 11 0 837 sh
[2497237.818907] [150766] 0 150766 1087 152 8 0 837 memsup
[2497237.827277] [150767] 0 150767 1088 87 8 0 837 cpu_sup
[2497237.835714] [151962] 0 151962 2898 155 11 0 837 inet_gethost
[2497237.844592] [151968] 0 151968 10396 325 25 0 837 inet_gethost
[2497237.853476] [152872] 0 152872 3505155 50904 557 0 837 beam.smp
[2497237.862000] [152942] 0 152942 1089 131 7 0 837 erl_child_setup
[2497237.871135] [154786] 0 154786 2898 114 11 0 837 inet_gethost
[2497237.880015] [154793] 0 154793 2898 86 11 0 837 inet_gethost
[2497237.888895] [154844] 0 154844 2924 342 12 0 837 sh
[2497237.896900] [154849] 0 154849 1087 148 8 0 837 memsup
[2497237.905257] [154853] 0 154853 1088 87 8 0 837 cpu_sup
[2497237.913695] [155053] 0 155053 2742 835 9 0 837 godu
[2497237.921875] [155066] 0 155066 2924 342 12 0 837 sh
[2497237.929886] [155067] 0 155067 1109 372 7 0 837 godu
[2497237.938066] [155326] 0 155326 55254 1128 20 0 837 saslauthd-port
[2497237.947111] [155348] 0 155348 2156 195 10 0 837 sigar_port
[2497237.955825] [155493] 0 155493 2814 917 9 0 837 goport
[2497237.964175] [155499] 0 155499 884435 178213 827 0 837 projector
[2497237.972790] [155658] 0 155658 8980804 3643971 10672 0 837 memcached
[2497237.981408] [170847] 0 170847 7502 540 21 0 837 ntpd
[2497237.989583] [171002] 0 171002 98079 72694 177 0 837 snmpd
[2497237.997858] [171110] 0 171110 3067 502 11 0 837 loopcouchbaseAl
[2497238.006988] [171131] 0 171131 1111 156 8 0 837 tail
[2497238.015169] [179997] 0 179997 10396 325 25 0 837 inet_gethost
[2497238.024079] [216518] 0 216518 2550 734 10 0 837 goport
[2497238.032431] [216529] 0 216529 191353 3542 46 0 837 goxdcr
[2497238.040912] [250234] 0 250234 3095 638 12 0 837 bash
[2497238.049091] [65602] 0 65602 2550 854 9 0 837 goport
[2497238.057274] [65608] 0 65608 4240181 1477389 3545 0 837 indexer
[2497238.065540] [69035] 0 69035 2550 740 9 0 837 goport
[2497238.073719] [69044] 0 69044 9689497 7800510 16464 0 837 cbq-engine
[2497238.082252] [69396] 0 69396 240957 3871 70 0 837 js-evaluator
[2497238.091223] [113638] 0 113638 3067 536 10 0 837 CouchbaseAlarms
[2497238.100376] [302272] 0 302272 3067 293 9 0 837 CouchbaseAlarms
[2497238.109511] [302274] 0 302274 18825 839 39 0 837 curl
[2497238.117693] [302275] 0 302275 3342 182 12 0 837 sed
[2497238.125784] [302276] 0 302276 2285 182 10 0 837 grep
[2497238.133967] [302278] 0 302278 2285 183 10 0 837 grep
[2497238.142148] [302279] 0 302279 3342 181 12 0 837 sed
[2497238.150241] [302280] 0 302280 2285 183 10 0 837 grep
[2497238.158423] [302281] 0 302281 3342 181 11 0 837 sed
[2497238.166521] [302282] 0 302282 2493 194 10 0 837 awk
[2497238.174658] Memory cgroup out of memory: Kill process 128809 (cbq-engine) score 1307 or sacrifice child
[2497238.184305] Killed process 69396 (js-evaluator), UID 0, total-vm:963828kB, anon-rss:5472kB, file-rss:10012kB, shmem-rss:0kB
Hello!
Kubernetes pods are typically killed based on the resources in use at the time that a reclamation is required. So while Couchbase may be the most memory hungry application running, it may not be what is causing the OOM kill. Check what other pods are being scheduled on the node and verify that that node has all of the necessary resources to run these pods.
Thanks,
Justin Ashworth