native btree_implemenation leaks memory [was: EPERF - couchbase ( memcached and beam.smp) go into 99% swap usage while running large cluster tests]

Farshid Ghods added a comment - 05/Feb/12 3:08 PM diags attached to MB-4761 have enough info about mem_used and swap usage for all nodes

Farshid Ghods added a comment - 05/Feb/12 3:26 PM swap used in the beginnng of the run            {swap_used,15880192}]},            {swap_used,17215488}]}, in the end            {swap_used,3396698112}]},            {swap_used,4294963200}]}, mem_used in the beginning : mem_used 2257279564 mem_used 3687230405 near the end mem_used 7549971323 mem_used 7549971323 10.140.5.80-8091-diag.txt

Steve Yen added a comment - 05/Feb/12 4:37 PM I am tempted to mark this as a duplicate of MB-4761 . (It's also possibly related to CBSE-100) While watching the live perf cluster (which Farshid provided), I see fill rate is 45K items/sec. Drain rate 20K items/sec. Already disk queue is >9M items, probably around 45 minutes into the "run" part of the perf test (which comes after the initial "load").

Steve Yen added a comment - 05/Feb/12 4:45 PM swap graph... https://skitch.com/steveyen/g67m2/couchbase-console-2.0.0

Farshid Ghods added a comment - 05/Feb/12 4:47 PM swap usage increased by 100 MB in a few minutes. ( increasing 1 MB per second ) beam.smp mem usage is bouncing back and forth between 4.8-4.9 GB ( 6611m 4.9g) and memcached is steadily at "8937m 8.3g" isn't 5 GB too much for beam.smp ? top - 22:43:15 up 2 days, 21:20, 1 user, load average: 2.66, 2.54, 1.86 Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie Cpu(s): 16.6%us, 15.0%sy, 0.0%ni, 51.4%id, 15.5%wa, 0.0%hi, 1.3%si, 0.1%st Mem: 15378216k total, 15295508k used, 82708k free, 10108k buffers Swap: 4194300k total, 374908k used, 3819392k free, 1220736k cached  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  6604 couchbas 20 0 6616m 4.9g 1920 S 177.2 33.2 79:34.89 beam.smp  6926 couchbas 20 0 8937m 8.4g 1380 S 29.3 57.1 17:20.57 memcached  351 root 20 0 0 0 0 S 2.0 0.0 47:30.73 kswapd0  1091 root 20 0 0 0 0 S 0.7 0.0 5:43.33 kjournald 14934 ec2-user 20 0 15052 1180 904 R 0.3 0.0 0:00.09 top

Farshid Ghods added a comment - 05/Feb/12 4:57 PM more info from Aliaksey beam seems to grow recently on those nodes. Still waiting more data. root@beta :~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"   PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND     2 2 TS - 0 19 0 0 S 0 0 kthreadd     3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0     4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0     5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0     6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1     7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1     8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1     9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2    10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2    11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2    12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3    13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3    14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3    15 15 TS - 0 19 0 0 S 0 0 \_ events/0    16 16 TS - 0 19 0 0 S 0 0 \_ events/1    17 17 TS - 0 19 0 0 S 0 0 \_ events/2    18 18 TS - 0 19 0 0 S 0 0 \_ events/3    19 19 TS - 0 19 0 0 S 0 0 \_ cpuset    20 20 TS - 0 19 0 0 S 0 0 \_ khelper   144 144 TS - 0 19 0 0 S 0 0 \_ netns   145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr   150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch   151 151 TS - 0 19 0 0 S 0 0 \_ xenbus   202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers   204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default   205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0   206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1   207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2   208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3   210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0   211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1   212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2   213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3   220 220 TS - 0 19 0 0 S 0 0 \_ kseriod   350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd   351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0   352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd   405 405 TS - 0 19 0 0 S 0 0 \_ aio/0   406 406 TS - 0 19 0 0 S 0 0 \_ aio/1   407 407 TS - 0 19 0 0 S 0 0 \_ aio/2   408 408 TS - 0 19 0 0 S 0 0 \_ aio/3   411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0   412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1   413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2   414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3   431 431 TS - 0 19 0 0 S 0 0 \_ khvcd   521 521 TS - 0 19 0 0 S 0 0 \_ kstriped   758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8   759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   795 795 TS - 0 19 0 0 S 0 0 \_ khubd  1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald  1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald  1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd 30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1 31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112     1 1 TS - 0 19 19244 192 Ss 3464 178 init   845 845 TS - -4 23 10684 4 S<s 977 2 udevd  1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd  1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd  1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient  1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd  1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd  1392 1392 TS - 0 19 9180 368 Ss 887392 897 irqbalance  1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon  1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd  4116 4116 TS - 0 19 97768 1632 Ss 4959 15 \_ sshd  4118 4118 TS - 0 19 97768 1620 S 16245 0 \_ sshd  1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd  1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail  1559 1559 TS - 0 19 117168 324 Ss 26036 587 crond  1569 1569 TS - 0 19 23604 0 Ss 222 41 atd  1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty  1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty  1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty  1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty  1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty  1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty  1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty  1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd  1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server 31088 31088 TS - 0 19 10856 284 S 132 2 epmd 31103 31103 TS - 0 19 7363684 5469980 Sl 18044454 649 beam.smp 31146 31146 TS - 0 19 106092 720 Ss 2649 3 \_ sh 31147 31147 TS - 0 19 4084 304 Ss 802 1 \_ memsup 31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup 31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost 31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost 31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi 31422 31422 TS - 0 19 9158384 8379440 Ssl 2335738 25477 \_ memcached 31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port  8429 8429 TS - 0 19 106088 1196 Ss 810 5 \_ sh  8430 8430 TS - 0 19 106032 960 R 384 2 \_ ps root@beta :~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"   PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND     2 2 TS - 0 19 0 0 S 0 0 kthreadd     3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0     4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0     5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0     6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1     7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1     8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1     9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2    10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2    11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2    12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3    13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3    14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3    15 15 TS - 0 19 0 0 S 0 0 \_ events/0    16 16 TS - 0 19 0 0 S 0 0 \_ events/1    17 17 TS - 0 19 0 0 S 0 0 \_ events/2    18 18 TS - 0 19 0 0 S 0 0 \_ events/3    19 19 TS - 0 19 0 0 S 0 0 \_ cpuset    20 20 TS - 0 19 0 0 S 0 0 \_ khelper   144 144 TS - 0 19 0 0 S 0 0 \_ netns   145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr   150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch   151 151 TS - 0 19 0 0 S 0 0 \_ xenbus   202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers   204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default   205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0   206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1   207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2   208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3   210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0   211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1   212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2   213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3   220 220 TS - 0 19 0 0 S 0 0 \_ kseriod   350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd   351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0   352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd   405 405 TS - 0 19 0 0 S 0 0 \_ aio/0   406 406 TS - 0 19 0 0 S 0 0 \_ aio/1   407 407 TS - 0 19 0 0 S 0 0 \_ aio/2   408 408 TS - 0 19 0 0 S 0 0 \_ aio/3   411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0   412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1   413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2   414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3   431 431 TS - 0 19 0 0 S 0 0 \_ khvcd   521 521 TS - 0 19 0 0 S 0 0 \_ kstriped   758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8   759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit   795 795 TS - 0 19 0 0 S 0 0 \_ khubd  1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald  1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald  1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd 30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1 31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112     1 1 TS - 0 19 19244 192 Ss 3464 178 init   845 845 TS - -4 23 10684 4 S<s 977 2 udevd  1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd  1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd  1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient  1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd  1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd  1392 1392 TS - 0 19 9180 368 Ss 887692 897 irqbalance  1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon  1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd  4116 4116 TS - 0 19 97768 1632 Ss 5184 15 \_ sshd  4118 4118 TS - 0 19 97768 1620 S 17180 0 \_ sshd  1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd  1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail  1559 1559 TS - 0 19 117168 320 Ss 26044 587 crond  1569 1569 TS - 0 19 23604 0 Ss 222 41 atd  1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty  1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty  1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty  1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty  1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty  1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty  1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty  1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd  1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server 31088 31088 TS - 0 19 10856 284 S 132 2 epmd 31103 31103 TS - 0 19 7391980 5483148 Sl 18318137 662 beam.smp 31146 31146 TS - 0 19 106092 720 Ss 2670 3 \_ sh 31147 31147 TS - 0 19 4084 304 Ss 804 1 \_ memsup 31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup 31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost 31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost 31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi 31422 31422 TS - 0 19 9158384 8319420 Ssl 2343827 28477 \_ memcached 31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port  8682 8682 TS - 0 19 106088 1200 Ss 817 1 \_ sh  8683 8683 TS - 0 19 106032 960 R 387 0 \_ ps root@beta :~#

Aleksey Kondratenko added a comment - 05/Feb/12 5:16 PM Reproduced locally by continuously SETting same 200k items. With default btree_implementation setting beam usage steadily grows. With erlang btree_implementation everything is fine.

Aaron Miller added a comment - 06/Feb/12 12:11 AM http://review.couchbase.org/#change,13018 fixes the leaks I was able to track down. Looks like they got introduced when I was adding checking for allocation failures.

Thuan Nguyen added a comment - 06/Feb/12 12:00 PM Integrated in github-couchdb-preview #325 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/325/ ])      MB-4763 Fix memory leaks in btree NIF (Revision 25dd55abbbd38e28294de9bece82994d4f8b741d)      Result = SUCCESS Aaron Miller : Files : * src/couchdb/priv/btree_nif/couch_btree.c * src/couchdb/priv/btree_nif/couch_btree_nif.c

Steve Yen added a comment - 06/Feb/12 7:11 PM Aaron, please go ahead and resolve issues when your fix is up on gerrit. Thx!