Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 2.0-developer-preview-4
-
Fix Version/s: 2.0-developer-preview-4
-
Component/s: view-engine
-
Security Level: Public
-
Labels:
-
Environment:build 631
Description
have observed this on recent large cluster performance test runs
the performance boxes have 4GB of swap space and during the run we seem to be using all the swap after a while and eventually OOM killer kicks in and terminates memcached.exe process as observed inMB-4761
the performance boxes have 4GB of swap space and during the run we seem to be using all the swap after a while and eventually OOM killer kicks in and terminates memcached.exe process as observed in
-
- mem.history.txt
- 05/Feb/12 3:26 PM
- 14 kB
- Farshid Ghods
-
- swap.history.txt
- 05/Feb/12 3:26 PM
- 203 kB
- Farshid Ghods
-
- Screen shot 2012-02-05 at 2.48.13 PM.png
- 67 kB
- 05/Feb/12 4:48 PM
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Hide
Farshid Ghods
added a comment -
swap used in the beginnng of the run
{swap_used,15880192}]},
{swap_used,17215488}]},
in the end
{swap_used,3396698112}]},
{swap_used,4294963200}]},
mem_used in the beginning :
mem_used 2257279564
mem_used 3687230405
near the end
mem_used 7549971323
mem_used 7549971323
10.140.5.80-8091-diag.txt
{swap_used,15880192}]},
{swap_used,17215488}]},
in the end
{swap_used,3396698112}]},
{swap_used,4294963200}]},
mem_used in the beginning :
mem_used 2257279564
mem_used 3687230405
near the end
mem_used 7549971323
mem_used 7549971323
10.140.5.80-8091-diag.txt
Show
Farshid Ghods
added a comment - swap used in the beginnng of the run
{swap_used,15880192}]},
{swap_used,17215488}]},
in the end
{swap_used,3396698112}]},
{swap_used,4294963200}]},
mem_used in the beginning :
mem_used 2257279564
mem_used 3687230405
near the end
mem_used 7549971323
mem_used 7549971323
10.140.5.80-8091-diag.txt
Hide
Steve Yen
added a comment -
I am tempted to mark this as a duplicate of MB-4761. (It's also possibly related to CBSE-100)
While watching the live perf cluster (which Farshid provided), I see fill rate is 45K items/sec. Drain rate 20K items/sec. Already disk queue is >9M items, probably around 45 minutes into the "run" part of the perf test (which comes after the initial "load").
While watching the live perf cluster (which Farshid provided), I see fill rate is 45K items/sec. Drain rate 20K items/sec. Already disk queue is >9M items, probably around 45 minutes into the "run" part of the perf test (which comes after the initial "load").
Show
Steve Yen
added a comment - I am tempted to mark this as a duplicate of MB-4761 . (It's also possibly related to CBSE-100)
While watching the live perf cluster (which Farshid provided), I see fill rate is 45K items/sec. Drain rate 20K items/sec. Already disk queue is >9M items, probably around 45 minutes into the "run" part of the perf test (which comes after the initial "load").
Hide
Farshid Ghods
added a comment -
swap usage increased by 100 MB in a few minutes. ( increasing 1 MB per second )
beam.smp mem usage is bouncing back and forth between 4.8-4.9 GB ( 6611m 4.9g)
and memcached is steadily at "8937m 8.3g"
isn't 5 GB too much for beam.smp ?
top - 22:43:15 up 2 days, 21:20, 1 user, load average: 2.66, 2.54, 1.86
Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.6%us, 15.0%sy, 0.0%ni, 51.4%id, 15.5%wa, 0.0%hi, 1.3%si, 0.1%st
Mem: 15378216k total, 15295508k used, 82708k free, 10108k buffers
Swap: 4194300k total, 374908k used, 3819392k free, 1220736k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6604 couchbas 20 0 6616m 4.9g 1920 S 177.2 33.2 79:34.89 beam.smp
6926 couchbas 20 0 8937m 8.4g 1380 S 29.3 57.1 17:20.57 memcached
351 root 20 0 0 0 0 S 2.0 0.0 47:30.73 kswapd0
1091 root 20 0 0 0 0 S 0.7 0.0 5:43.33 kjournald
14934 ec2-user 20 0 15052 1180 904 R 0.3 0.0 0:00.09 top
beam.smp mem usage is bouncing back and forth between 4.8-4.9 GB ( 6611m 4.9g)
and memcached is steadily at "8937m 8.3g"
isn't 5 GB too much for beam.smp ?
top - 22:43:15 up 2 days, 21:20, 1 user, load average: 2.66, 2.54, 1.86
Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.6%us, 15.0%sy, 0.0%ni, 51.4%id, 15.5%wa, 0.0%hi, 1.3%si, 0.1%st
Mem: 15378216k total, 15295508k used, 82708k free, 10108k buffers
Swap: 4194300k total, 374908k used, 3819392k free, 1220736k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6604 couchbas 20 0 6616m 4.9g 1920 S 177.2 33.2 79:34.89 beam.smp
6926 couchbas 20 0 8937m 8.4g 1380 S 29.3 57.1 17:20.57 memcached
351 root 20 0 0 0 0 S 2.0 0.0 47:30.73 kswapd0
1091 root 20 0 0 0 0 S 0.7 0.0 5:43.33 kjournald
14934 ec2-user 20 0 15052 1180 904 R 0.3 0.0 0:00.09 top
Show
Farshid Ghods
added a comment - swap usage increased by 100 MB in a few minutes. ( increasing 1 MB per second )
beam.smp mem usage is bouncing back and forth between 4.8-4.9 GB ( 6611m 4.9g)
and memcached is steadily at "8937m 8.3g"
isn't 5 GB too much for beam.smp ?
top - 22:43:15 up 2 days, 21:20, 1 user, load average: 2.66, 2.54, 1.86
Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.6%us, 15.0%sy, 0.0%ni, 51.4%id, 15.5%wa, 0.0%hi, 1.3%si, 0.1%st
Mem: 15378216k total, 15295508k used, 82708k free, 10108k buffers
Swap: 4194300k total, 374908k used, 3819392k free, 1220736k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6604 couchbas 20 0 6616m 4.9g 1920 S 177.2 33.2 79:34.89 beam.smp
6926 couchbas 20 0 8937m 8.4g 1380 S 29.3 57.1 17:20.57 memcached
351 root 20 0 0 0 0 S 2.0 0.0 47:30.73 kswapd0
1091 root 20 0 0 0 0 S 0.7 0.0 5:43.33 kjournald
14934 ec2-user 20 0 15052 1180 904 R 0.3 0.0 0:00.09 top
Hide
Farshid Ghods
added a comment -
more info from Aliaksey
beam seems to grow recently on those nodes. Still waiting more data.
root@beta:~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887392 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 4959 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 16245 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 324 Ss 26036 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7363684 5469980 Sl 18044454 649 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2649 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 802 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8379440 Ssl 2335738 25477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8429 8429 TS - 0 19 106088 1196 Ss 810 5 \_ sh
8430 8430 TS - 0 19 106032 960 R 384 2 \_ ps
root@beta:~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887692 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 5184 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 17180 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 320 Ss 26044 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7391980 5483148 Sl 18318137 662 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2670 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 804 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8319420 Ssl 2343827 28477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8682 8682 TS - 0 19 106088 1200 Ss 817 1 \_ sh
8683 8683 TS - 0 19 106032 960 R 387 0 \_ ps
root@beta:~#
beam seems to grow recently on those nodes. Still waiting more data.
root@beta:~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887392 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 4959 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 16245 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 324 Ss 26036 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7363684 5469980 Sl 18044454 649 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2649 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 802 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8379440 Ssl 2335738 25477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8429 8429 TS - 0 19 106088 1196 Ss 810 5 \_ sh
8430 8430 TS - 0 19 106032 960 R 384 2 \_ ps
root@beta:~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887692 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 5184 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 17180 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 320 Ss 26044 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7391980 5483148 Sl 18318137 662 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2670 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 804 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8319420 Ssl 2343827 28477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8682 8682 TS - 0 19 106088 1200 Ss 817 1 \_ sh
8683 8683 TS - 0 19 106032 960 R 387 0 \_ ps
root@beta:~#
Show
Farshid Ghods
added a comment - more info from Aliaksey
beam seems to grow recently on those nodes. Still waiting more data.
root@beta :~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887392 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 4959 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 16245 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 324 Ss 26036 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7363684 5469980 Sl 18044454 649 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2649 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 802 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8379440 Ssl 2335738 25477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8429 8429 TS - 0 19 106088 1196 Ss 810 5 \_ sh
8430 8430 TS - 0 19 106032 960 R 384 2 \_ ps
root@beta :~# wget -O- -q --user=Administrator --password=password --post-data='os:cmd("ps -eo pid,tid,class,rtprio,ni,pri,vsize,rss,stat,min_flt,maj_flt,comm --forest").' http://23.20.47.84:8091/diag/eval | ruby -e "puts eval(STDIN.read)"
PID TID CLS RTPRIO NI PRI VSZ RSS STAT MINFL MAJFL COMMAND
2 2 TS - 0 19 0 0 S 0 0 kthreadd
3 3 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/0
4 4 FF 99 - 139 0 0 S 0 0 \_ migration/0
5 5 FF 99 - 139 0 0 S 0 0 \_ watchdog/0
6 6 FF 99 - 139 0 0 S 0 0 \_ migration/1
7 7 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/1
8 8 FF 99 - 139 0 0 S 0 0 \_ watchdog/1
9 9 FF 99 - 139 0 0 S 0 0 \_ migration/2
10 10 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/2
11 11 FF 99 - 139 0 0 S 0 0 \_ watchdog/2
12 12 FF 99 - 139 0 0 S 0 0 \_ migration/3
13 13 TS - 0 19 0 0 S 0 0 \_ ksoftirqd/3
14 14 FF 99 - 139 0 0 S 0 0 \_ watchdog/3
15 15 TS - 0 19 0 0 S 0 0 \_ events/0
16 16 TS - 0 19 0 0 S 0 0 \_ events/1
17 17 TS - 0 19 0 0 S 0 0 \_ events/2
18 18 TS - 0 19 0 0 S 0 0 \_ events/3
19 19 TS - 0 19 0 0 S 0 0 \_ cpuset
20 20 TS - 0 19 0 0 S 0 0 \_ khelper
144 144 TS - 0 19 0 0 S 0 0 \_ netns
145 145 TS - 0 19 0 0 S 0 0 \_ async/mgr
150 150 TS - 0 19 0 0 S 0 0 \_ xenwatch
151 151 TS - 0 19 0 0 S 0 0 \_ xenbus
202 202 TS - 0 19 0 0 S 0 0 \_ sync_supers
204 204 TS - 0 19 0 0 S 0 0 \_ bdi-default
205 205 TS - 0 19 0 0 S 0 0 \_ kintegrityd/0
206 206 TS - 0 19 0 0 S 0 0 \_ kintegrityd/1
207 207 TS - 0 19 0 0 S 0 0 \_ kintegrityd/2
208 208 TS - 0 19 0 0 S 0 0 \_ kintegrityd/3
210 210 TS - 0 19 0 0 S 0 0 \_ kblockd/0
211 211 TS - 0 19 0 0 S 0 0 \_ kblockd/1
212 212 TS - 0 19 0 0 S 0 0 \_ kblockd/2
213 213 TS - 0 19 0 0 S 0 0 \_ kblockd/3
220 220 TS - 0 19 0 0 S 0 0 \_ kseriod
350 350 TS - 0 19 0 0 S 0 0 \_ khungtaskd
351 351 TS - 0 19 0 0 S 0 0 \_ kswapd0
352 352 TS - 5 14 0 0 SN 0 0 \_ ksmd
405 405 TS - 0 19 0 0 S 0 0 \_ aio/0
406 406 TS - 0 19 0 0 S 0 0 \_ aio/1
407 407 TS - 0 19 0 0 S 0 0 \_ aio/2
408 408 TS - 0 19 0 0 S 0 0 \_ aio/3
411 411 TS - 0 19 0 0 S 0 0 \_ crypto/0
412 412 TS - 0 19 0 0 S 0 0 \_ crypto/1
413 413 TS - 0 19 0 0 S 0 0 \_ crypto/2
414 414 TS - 0 19 0 0 S 0 0 \_ crypto/3
431 431 TS - 0 19 0 0 S 0 0 \_ khvcd
521 521 TS - 0 19 0 0 S 0 0 \_ kstriped
758 758 TS - 0 19 0 0 S 0 0 \_ jbd2/xvda1-8
759 759 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
760 760 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
761 761 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
762 762 TS - 0 19 0 0 S 0 0 \_ ext4-dio-unwrit
795 795 TS - 0 19 0 0 S 0 0 \_ khubd
1104 1104 TS - 0 19 0 0 S 0 0 \_ kjournald
1105 1105 TS - 0 19 0 0 S 0 0 \_ kjournald
1140 1140 TS - 0 19 0 0 S 0 0 \_ kauditd
30120 30120 TS - 0 19 0 0 S 0 0 \_ flush-202:1
31251 31251 TS - 0 19 0 0 S 0 0 \_ flush-202:112
1 1 TS - 0 19 19244 192 Ss 3464 178 init
845 845 TS - -4 23 10684 4 S<s 977 2 udevd
1178 1178 TS - -2 21 10680 4 S< 114 1 \_ udevd
1179 1179 TS - -2 21 10680 4 S< 116 0 \_ udevd
1329 1329 TS - 0 19 9144 332 Ss 890 77 dhclient
1366 1366 TS - -4 23 89080 100 S<sl 4320 541 auditd
1381 1381 TS - 0 19 248904 432 Sl 7164 977 rsyslogd
1392 1392 TS - 0 19 9180 368 Ss 887692 897 irqbalance
1403 1403 TS - 0 19 21408 256 Ss 2392 142 dbus-daemon
1502 1502 TS - 0 19 61928 284 Ss 1570 94 sshd
4116 4116 TS - 0 19 97768 1632 Ss 5184 15 \_ sshd
4118 4118 TS - 0 19 97768 1620 S 17180 0 \_ sshd
1528 1528 TS - 0 19 30172 236 Ss 5006 864 ntpd
1543 1543 TS - 0 19 84312 400 Ss 9517 1955 sendmail
1559 1559 TS - 0 19 117168 320 Ss 26044 587 crond
1569 1569 TS - 0 19 23604 0 Ss 222 41 atd
1594 1594 TS - 0 19 4104 8 Ss+ 549 2 agetty
1595 1595 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1599 1599 TS - 0 19 4092 4 Ss+ 537 2 mingetty
1602 1602 TS - 0 19 4092 4 Ss+ 538 1 mingetty
1607 1607 TS - 0 19 4092 4 Ss+ 539 1 mingetty
1610 1610 TS - 0 19 4092 4 Ss+ 536 2 mingetty
1615 1615 TS - 0 19 4092 4 Ss+ 537 1 mingetty
1616 1616 TS - 19 0 333292 2556 SN 32248 1691 yum-updatesd
1618 1618 TS - 19 0 9264 252 SN 1028 73 gam_server
31088 31088 TS - 0 19 10856 284 S 132 2 epmd
31103 31103 TS - 0 19 7391980 5483148 Sl 18318137 662 beam.smp
31146 31146 TS - 0 19 106092 720 Ss 2670 3 \_ sh
31147 31147 TS - 0 19 4084 304 Ss 804 1 \_ memsup
31148 31148 TS - 0 19 4080 0 Ss 586 1 \_ cpu_sup
31168 31168 TS - 0 19 10820 8 Ss 627 0 \_ inet_gethost
31169 31169 TS - 0 19 10820 188 S 92 0 | \_ inet_gethost
31421 31421 TS - 0 19 408620 1020 Ssl 1247 87 \_ moxi
31422 31422 TS - 0 19 9158384 8319420 Ssl 2343827 28477 \_ memcached
31423 31423 TS - 0 19 6276 212 Ss 284 3 \_ sigar_port
8682 8682 TS - 0 19 106088 1200 Ss 817 1 \_ sh
8683 8683 TS - 0 19 106032 960 R 387 0 \_ ps
root@beta :~#
Hide
Aleksey Kondratenko
added a comment -
Reproduced locally by continuously SETting same 200k items. With default btree_implementation setting beam usage steadily grows. With erlang btree_implementation everything is fine.
Show
Aleksey Kondratenko
added a comment - Reproduced locally by continuously SETting same 200k items. With default btree_implementation setting beam usage steadily grows. With erlang btree_implementation everything is fine.
Hide
Aaron Miller
added a comment -
http://review.couchbase.org/#change,13018 fixes the leaks I was able to track down. Looks like they got introduced when I was adding checking for allocation failures.
Show
Aaron Miller
added a comment - http://review.couchbase.org/#change,13018 fixes the leaks I was able to track down. Looks like they got introduced when I was adding checking for allocation failures.
Hide
Thuan Nguyen
added a comment -
Integrated in github-couchdb-preview #325 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/325/])
MB-4763 Fix memory leaks in btree NIF (Revision 25dd55abbbd38e28294de9bece82994d4f8b741d)
Result = SUCCESS
Aaron Miller :
Files :
* src/couchdb/priv/btree_nif/couch_btree.c
* src/couchdb/priv/btree_nif/couch_btree_nif.c
Result = SUCCESS
Aaron Miller :
Files :
* src/couchdb/priv/btree_nif/couch_btree.c
* src/couchdb/priv/btree_nif/couch_btree_nif.c
Show
Thuan Nguyen
added a comment - Integrated in github-couchdb-preview #325 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/325/ ])
MB-4763 Fix memory leaks in btree NIF (Revision 25dd55abbbd38e28294de9bece82994d4f8b741d)
Result = SUCCESS
Aaron Miller :
Files :
* src/couchdb/priv/btree_nif/couch_btree.c
* src/couchdb/priv/btree_nif/couch_btree_nif.c
MB-4761have enough info about mem_used and swap usage for all nodes