The problem in the moxi
Hello
I had a problem with couchbase.
Tell me how to solve the problem and how to analyze it.
Mar 9 09:40:26 host1 kernel: shrink_slab: nr=-229008293888
Mar 9 09:42:54 host1 kernel: INFO: task beam.smp:21742 blocked for more than 120 seconds.
Mar 9 09:42:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:42:54 host1 kernel: beam.smp D ffffffff801568f1 0 21742 1 21743 21741 (NOTLB)
Mar 9 09:42:54 host1 kernel: ffff81139945fef8 0000000000000082 ffff81139945fe78 ffffffff8002dfcc
Mar 9 09:42:54 host1 kernel: ffff81180f1b7480 000000000000000a ffff8114daa25820 ffff810c70113860
Mar 9 09:42:54 host1 kernel: 0026d170b8536b48 000000000037f3aa ffff8114daa25a08 000000028002a082
Mar 9 09:42:54 host1 kernel: Call Trace:
Mar 9 09:42:54 host1 kernel: [] __wake_up+0x38/0x4f
Mar 9 09:42:54 host1 kernel: [] __down_write_nested+0x7a/0x92
Mar 9 09:42:54 host1 kernel: [] sys_mmap_pgoff+0x55/0xac
Mar 9 09:42:54 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:42:54 host1 kernel:
Mar 9 09:42:54 host1 kernel: INFO: task beam.smp:21747 blocked for more than 120 seconds.
Mar 9 09:42:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:42:54 host1 kernel: beam.smp D ffffffff801568f1 0 21747 1 21748 21746 (NOTLB)
Mar 9 09:42:54 host1 kernel: ffff811569567e18 0000000000000082 ffff810c9a1a4680 000000000000000e
Mar 9 09:42:54 host1 kernel: ffff810c90962440 0000000000000009 ffff811710ee4860 ffff81012b93e7e0
Mar 9 09:42:54 host1 kernel: 0026d170b854126d 00000000000903fa ffff811710ee4a48 0000000180009007
Mar 9 09:42:54 host1 kernel: Call Trace:
Mar 9 09:42:54 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:42:54 host1 kernel: [] do_page_fault+0x414/0x842
Mar 9 09:42:54 host1 kernel: [] error_exit+0x0/0x84
Mar 9 09:42:54 host1 kernel:
Mar 9 09:44:54 host1 kernel: INFO: task beam.smp:21742 blocked for more than 120 seconds.
Mar 9 09:44:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:44:54 host1 kernel: beam.smp D ffffffff801568f1 0 21742 1 21743 21741 (NOTLB)
Mar 9 09:44:54 host1 kernel: ffff81139945fef8 0000000000000082 ffff81139945fe78 ffffffff8002dfcc
Mar 9 09:44:54 host1 kernel: ffff81180f1b7480 000000000000000a ffff8114daa25820 ffff810c70113860
Mar 9 09:44:54 host1 kernel: 0026d170b8536b48 000000000037f3aa ffff8114daa25a08 000000028002a082
Mar 9 09:44:54 host1 kernel: Call Trace:
Mar 9 09:44:54 host1 kernel: [] __wake_up+0x38/0x4f
Mar 9 09:44:54 host1 kernel: [] __down_write_nested+0x7a/0x92
Mar 9 09:44:54 host1 kernel: [] sys_mmap_pgoff+0x55/0xac
Mar 9 09:44:54 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:44:54 host1 kernel:
Mar 9 09:44:54 host1 kernel: INFO: task beam.smp:21747 blocked for more than 120 seconds.
Mar 9 09:44:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:44:54 host1 kernel: beam.smp D ffffffff801568f1 0 21747 1 21748 21746 (NOTLB)
Mar 9 09:44:54 host1 kernel: ffff811569567e18 0000000000000082 ffff810c9a1a4680 000000000000000e
Mar 9 09:44:54 host1 kernel: ffff810c90962440 0000000000000009 ffff811710ee4860 ffff81012b93e7e0
Mar 9 09:44:54 host1 kernel: 0026d170b854126d 00000000000903fa ffff811710ee4a48 0000000180009007
Mar 9 09:44:54 host1 kernel: Call Trace:
Mar 9 09:44:54 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:44:54 host1 kernel: [] do_page_fault+0x414/0x842
Mar 9 09:44:54 host1 kernel: [] error_exit+0x0/0x84
Mar 9 09:44:54 host1 kernel:
Mar 9 09:44:54 host1 kernel: INFO: task ps:5256 blocked for more than 120 seconds.
Mar 9 09:44:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:44:54 host1 kernel: ps D ffffffff801568f1 0 5256 5254 5257 (NOTLB)
Mar 9 09:44:54 host1 kernel: ffff8117404a7d98 0000000000000086 ffff810c6f7ea8c0 ffff8117efc5b810
Mar 9 09:44:54 host1 kernel: ffff8117404a7ea8 0000000000000007 ffff811771bba040 ffff81186fccd7a0
Mar 9 09:44:54 host1 kernel: 0026d17900c31564 0000000000196afc ffff811771bba228 0000000f00000000
Mar 9 09:44:54 host1 kernel: Call Trace:
Mar 9 09:44:54 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:44:54 host1 kernel: [] __access_process_vm+0x3c/0x171
Mar 9 09:44:54 host1 kernel: [] access_process_vm+0x46/0x60
Mar 9 09:44:54 host1 kernel: [] proc_pid_cmdline+0x69/0xf4
Mar 9 09:44:54 host1 kernel: [] proc_info_read+0x5f/0xb9
Mar 9 09:44:54 host1 kernel: [] vfs_read+0xcb/0x171
Mar 9 09:44:54 host1 kernel: [] sys_read+0x45/0x6e
Mar 9 09:44:54 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:44:54 host1 kernel:
Mar 9 09:44:54 host1 kernel: INFO: task ps:26307 blocked for more than 120 seconds.
Mar 9 09:44:54 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:44:54 host1 kernel: ps D ffffffff801568f1 0 26307 26306 26308 (NOTLB)
Mar 9 09:44:54 host1 kernel: ffff81149f467d98 0000000000000082 ffff810c6f7ea8c0 ffff8117efc5b810
Mar 9 09:44:54 host1 kernel: ffff81149f467ea8 0000000000000001 ffff8117494e1860 ffff81186fe08820
Mar 9 09:44:55 host1 kernel: 0026d186f13c2300 00000000006b3de2 ffff8117494e1a48 0000001500000000
Mar 9 09:44:55 host1 kernel: Call Trace:
Mar 9 09:44:55 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:44:55 host1 kernel: [] __access_process_vm+0x3c/0x171
Mar 9 09:44:55 host1 kernel: [] access_process_vm+0x46/0x60
Mar 9 09:44:55 host1 kernel: [] proc_pid_cmdline+0x69/0xf4
Mar 9 09:44:55 host1 kernel: [] proc_info_read+0x5f/0xb9
Mar 9 09:44:55 host1 kernel: [] vfs_read+0xcb/0x171
Mar 9 09:44:55 host1 kernel: [] sys_read+0x45/0x6e
Mar 9 09:44:55 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:44:55 host1 kernel:
Mar 9 09:46:55 host1 kernel: INFO: task beam.smp:21742 blocked for more than 120 seconds.
Mar 9 09:46:55 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:46:55 host1 kernel: beam.smp D ffffffff801568f1 0 21742 1 21743 21741 (NOTLB)
Mar 9 09:46:55 host1 kernel: ffff81139945fef8 0000000000000082 ffff81139945fe78 ffffffff8002dfcc
Mar 9 09:46:55 host1 kernel: ffff81180f1b7480 000000000000000a ffff8114daa25820 ffff810c70113860
Mar 9 09:46:55 host1 kernel: 0026d170b8536b48 000000000037f3aa ffff8114daa25a08 000000028002a082
Mar 9 09:46:55 host1 kernel: Call Trace:
Mar 9 09:46:55 host1 kernel: [] __wake_up+0x38/0x4f
Mar 9 09:46:55 host1 kernel: [] __down_write_nested+0x7a/0x92
Mar 9 09:46:55 host1 kernel: [] sys_mmap_pgoff+0x55/0xac
Mar 9 09:46:55 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:46:55 host1 kernel:
Mar 9 09:46:55 host1 kernel: INFO: task beam.smp:21747 blocked for more than 120 seconds.
Mar 9 09:46:55 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:46:55 host1 kernel: beam.smp D ffffffff801568f1 0 21747 1 21748 21746 (NOTLB)
Mar 9 09:46:55 host1 kernel: ffff811569567e18 0000000000000082 ffff810c9a1a4680 000000000000000e
Mar 9 09:46:55 host1 kernel: ffff810c90962440 0000000000000009 ffff811710ee4860 ffff81012b93e7e0
Mar 9 09:46:55 host1 kernel: 0026d170b854126d 00000000000903fa ffff811710ee4a48 0000000180009007
Mar 9 09:46:55 host1 kernel: Call Trace:
Mar 9 09:46:55 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:46:55 host1 kernel: [] do_page_fault+0x414/0x842
Mar 9 09:46:55 host1 kernel: [] error_exit+0x0/0x84
Mar 9 09:46:55 host1 kernel:
Mar 9 09:46:55 host1 kernel: INFO: task ps:5256 blocked for more than 120 seconds.
Mar 9 09:46:55 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:46:55 host1 kernel: ps D ffffffff801568f1 0 5256 5254 5257 (NOTLB)
Mar 9 09:46:55 host1 kernel: ffff8117404a7d98 0000000000000086 ffff810c6f7ea8c0 ffff8117efc5b810
Mar 9 09:46:55 host1 kernel: ffff8117404a7ea8 0000000000000007 ffff811771bba040 ffff81186fccd7a0
Mar 9 09:46:55 host1 kernel: 0026d17900c31564 0000000000196afc ffff811771bba228 0000000f00000000
Mar 9 09:46:55 host1 kernel: Call Trace:
Mar 9 09:46:55 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:46:55 host1 kernel: [] __access_process_vm+0x3c/0x171
Mar 9 09:46:55 host1 kernel: [] access_process_vm+0x46/0x60
Mar 9 09:46:55 host1 kernel: [] proc_pid_cmdline+0x69/0xf4
Mar 9 09:46:55 host1 kernel: [] proc_info_read+0x5f/0xb9
Mar 9 09:46:55 host1 kernel: [] vfs_read+0xcb/0x171
Mar 9 09:46:55 host1 kernel: [] sys_read+0x45/0x6e
Mar 9 09:46:55 host1 kernel: [] tracesys+0xd5/0xe0
Mar 9 09:46:55 host1 kernel:
Mar 9 09:46:55 host1 kernel: INFO: task ps:26307 blocked for more than 120 seconds.
Mar 9 09:46:55 host1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 9 09:46:55 host1 kernel: ps D ffffffff801568f1 0 26307 26306 26308 (NOTLB)
Mar 9 09:46:55 host1 kernel: ffff81149f467d98 0000000000000082 ffff810c6f7ea8c0 ffff8117efc5b810
Mar 9 09:46:55 host1 kernel: ffff81149f467ea8 0000000000000001 ffff8117494e1860 ffff81186fe08820
Mar 9 09:46:55 host1 kernel: 0026d186f13c2300 00000000006b3de2 ffff8117494e1a48 0000001500000000
Mar 9 09:46:55 host1 kernel: Call Trace:
Mar 9 09:46:55 host1 kernel: [] __down_read+0x7a/0x92
Mar 9 09:46:55 host1 kernel: [] __access_process_vm+0x3c/0x171
Mar 9 09:46:55 host1 kernel: [] access_process_vm+0x46/0x60
Mar 9 09:46:55 host1 kernel: [] proc_pid_cmdline+0x69/0xf4
Mar 9 09:46:55 host1 kernel: [] proc_info_read+0x5f/0xb9
Mar 9 09:46:55 host1 kernel: [] vfs_read+0xcb/0x171
Mar 9 09:46:55 host1 kernel: [] sys_read+0x45/0x6e
Mar 9 09:46:55 host1 kernel: [] tracesys+0xd5/0xe0
The following bug has been created for this:
http://www.couchbase.com/issues/browse/MB-7948
Regards
The following bug has been created for this:
http://www.couchbase.com/issues/browse/MB-7948
Regards
Tug
@tgrall