Couchbase Server

[Windows] severe timeouts in windows cluster with 2 buckets and 1 ddoc per bucket causes rebalance and other failures

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Duplicate
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.1
  • Component/s: ns_server
  • Security Level: Public

Description

this is a master bug that we are creating and linking other windows related failure bugs to this one
  1. Zip Archive
    disconnects.zip
    25/Feb/13 12:32 AM
    175 kB
    Sriram Melkote

Issue Links

blocks

Bug - A problem which impairs or prevents the functions of the product. MB-7902 [windows] Rebalance exited with reason {detected_nodes_change, {ns_node_disco_events,

  • Blocker - Blocks development and/or testing work, production could not run.
  • Open - The issue is open and ready for the assignee to start work on it.

Bug - A problem which impairs or prevents the functions of the product. MB-7644 [system test] rebalance failed with error Rebalance exited with reason {{badmatch,false}

  • Blocker - Blocks development and/or testing work, production could not run.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Bug - A problem which impairs or prevents the functions of the product. MB-7595 [system test] rebalance failed with error "Resetting rebalance status since it's not really running"

  • Major - Major loss of function.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Bug - A problem which impairs or prevents the functions of the product. MB-7596 [system test] memsup crashed due to os_mon_sysinfo timeout

  • Major - Major loss of function.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Bug - A problem which impairs or prevents the functions of the product. MB-7590 [system test] rebalance failed due to buckets shutdown in orchestrator node

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.
depends on

Bug - A problem which impairs or prevents the functions of the product. MB-7920 [windows] Rebalancing out the last node causes the cluster to be reinitialized

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.
is duplicated by

Bug - A problem which impairs or prevents the functions of the product. MB-7902 [windows] Rebalance exited with reason {detected_nodes_change, {ns_node_disco_events,

  • Blocker - Blocks development and/or testing work, production could not run.
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Thuan Nguyen added a comment -
Environment:
** 4 physical servers. Each server has 32GB RAM, 4 core CPU and 2 regular spin disks

10.2.1.61
10.2.1.62
10.2.1.63
10.2.1.64

Cluster setup:
** Create cluster with 2 nodes (10.2.1.63 [master] and 10.2.1.64)
** Data path on c:/data and view path on d:/view or e:/view
** Couchbase RAM quota: 28GB
** Create 2 buckets, 14GB default bucket and 10GB sasl bucket (each bucket has 1 replica and index enable)
** Each bucket has one doc and 2 views
** Load 30 million items to default bucket and 20 million items to sasl bucket to let the active resident ratio down to around 70%
(as spec in http://hub.internal.couchbase.com/confluence/display/QA/views-test but I modify it to fix in 2 nodes cluster)

** Run access phase in 3 hours

Then running RB-1:
** Add node 10.2.1.61 to cluster and rebalance. See sasl bucket rebalance first.
** Monitor rebalance process. Seeing rebalance on node 61 reach 50% white 45% on other 2 nodes (63 and 64)
** Then few minutes later, rebalance failed with error "Resetting rebalance status since it's not really running"

[user:info,2013-02-02T1:58:40.905,ns_1@10.2.1.61:ns_config<0.3600.6>:ns_janitor:maybe_stop_rebalance_status:147]Resetting rebalance status since it's not really running


** On diags of node 61, node 61 could not talk to orchestrator, so it took over orchestrator

[ns_server:error,2013-02-02T1:58:27.895,ns_1@10.2.1.61:<0.5315.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:27.895,ns_1@10.2.1.61:<0.499.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:28.004,ns_1@10.2.1.61:<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:28.066,ns_1@10.2.1.61:<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:32.029,ns_1@10.2.1.61:<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:32.434,ns_1@10.2.1.61:<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:34.494,ns_1@10.2.1.61:<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[ns_server:error,2013-02-02T1:58:34.556,ns_1@10.2.1.61:<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,
                                {timeout,
                                 {gen_fsm,sync_send_event,
                                  [{global,ns_orchestrator},
                                   rebalance_progress,2000]}}}
[user:info,2013-02-02T1:58:36.756,ns_1@10.2.1.61:mb_master<0.14711.6>:mb_master:handle_info:219]Haven't heard from a higher priority node or a master, so I'm taking over.
[ns_server:debug,2013-02-02T1:58:36.756,ns_1@10.2.1.61:mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_fsm, ns_orchestrator, [], []): monitoring <20150.3131.0> from 'ns_1@10.2.1.61'
[ns_server:debug,2013-02-02T1:58:36.756,ns_1@10.2.1.61:mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_server, ns_tick, [], []): monitoring <20150.3132.0> from 'ns_1@10.2.1.61'
[ns_server:debug,2013-02-02T1:58:36.756,ns_1@10.2.1.61:mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_server, auto_failover, [], []): monitoring <20150.3134.0> from 'ns_1@10.2.1.61'
[error_logger:info,2013-02-02T1:58:36.756,ns_1@10.2.1.61:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]

** In diags node 63 (old orchestrator) I saw a lot vbucket_mover crashed around rebalance failed

========================CRASH REPORT=========================
  crasher:
    initial call: ns_single_vbucket_mover:mover/6
    pid: <0.15041.759>
    registered_name: []
    exception exit: {unexpected_exit,{'EXIT',<0.7018.476>,shutdown}}
      in function ns_single_vbucket_mover:spawn_and_wait/1
      in call from ns_single_vbucket_mover:mover_inner/6
      in call from misc:try_with_maybe_ignorant_after/2
      in call from ns_single_vbucket_mover:mover/6
    ancestors: [<0.7018.476>,<0.5453.476>]
    messages: [{'EXIT',<0.7018.476>,shutdown}]
    links: [<0.7018.476>]
    dictionary: [{cleanup_list,[<0.15227.759>,<0.16464.759>]}]
    trap_exit: true
    status: running
    heap_size: 2584
    stack_size: 24
    reductions: 5950
  neighbours:


** Time stamp of this test:

Bucket "sasl" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.63 20:17:01 - Fri Feb 1, 2013

Started rebalancing bucket sasl ns_rebalancer000 ns_1@10.2.1.63 20:16:58 - Fri Feb 1, 2013
Starting rebalance, KeepNodes = ['ns_1@10.2.1.64','ns_1@10.2.1.63',
'ns_1@10.2.1.61'], EjectNodes = [] ns_orchestrator004 ns_1@10.2.1.63 20:16:58 - Fri Feb 1, 2013

Rebalance failed at this phase RB-1 right before complete rebalance of first bucket.

Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.2.1.61 01:58:40 - Sat Feb 2, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 01:58:36 - Sat Feb 2, 2013

Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 02:12:25 - Sat Feb 2, 2013


Link to manifest file http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-144-rel.setup.exe.manifest.xml

Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201301/3nodes-phy-servers-201-144-reb-stopped-by-janitor-20130201-183832.tgz
Show
Thuan Nguyen added a comment - Environment: ** 4 physical servers. Each server has 32GB RAM, 4 core CPU and 2 regular spin disks 10.2.1.61 10.2.1.62 10.2.1.63 10.2.1.64 Cluster setup: ** Create cluster with 2 nodes (10.2.1.63 [master] and 10.2.1.64) ** Data path on c:/data and view path on d:/view or e:/view ** Couchbase RAM quota: 28GB ** Create 2 buckets, 14GB default bucket and 10GB sasl bucket (each bucket has 1 replica and index enable) ** Each bucket has one doc and 2 views ** Load 30 million items to default bucket and 20 million items to sasl bucket to let the active resident ratio down to around 70% (as spec in http://hub.internal.couchbase.com/confluence/display/QA/views-test but I modify it to fix in 2 nodes cluster) ** Run access phase in 3 hours Then running RB-1: ** Add node 10.2.1.61 to cluster and rebalance. See sasl bucket rebalance first. ** Monitor rebalance process. Seeing rebalance on node 61 reach 50% white 45% on other 2 nodes (63 and 64) ** Then few minutes later, rebalance failed with error "Resetting rebalance status since it's not really running" [user:info,2013-02-02T1:58:40.905, ns_1@10.2.1.61 :ns_config<0.3600.6>:ns_janitor:maybe_stop_rebalance_status:147]Resetting rebalance status since it's not really running ** On diags of node 61, node 61 could not talk to orchestrator, so it took over orchestrator [ns_server:error,2013-02-02T1:58:27.895, ns_1@10.2.1.61 :<0.5315.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:27.895, ns_1@10.2.1.61 :<0.499.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:28.004, ns_1@10.2.1.61 :<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:28.066, ns_1@10.2.1.61 :<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:32.029, ns_1@10.2.1.61 :<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:32.434, ns_1@10.2.1.61 :<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:34.494, ns_1@10.2.1.61 :<0.7772.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error,2013-02-02T1:58:34.556, ns_1@10.2.1.61 :<0.6957.14>:ns_orchestrator:rebalance_progress:176]Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [user:info,2013-02-02T1:58:36.756, ns_1@10.2.1.61 :mb_master<0.14711.6>:mb_master:handle_info:219]Haven't heard from a higher priority node or a master, so I'm taking over. [ns_server:debug,2013-02-02T1:58:36.756, ns_1@10.2.1.61 :mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_fsm, ns_orchestrator, [], []): monitoring <20150.3131.0> from ' ns_1@10.2.1.61 ' [ns_server:debug,2013-02-02T1:58:36.756, ns_1@10.2.1.61 :mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_server, ns_tick, [], []): monitoring <20150.3132.0> from ' ns_1@10.2.1.61 ' [ns_server:debug,2013-02-02T1:58:36.756, ns_1@10.2.1.61 :mb_master_sup<0.9505.14>:misc:start_singleton:854]start_singleton(gen_server, auto_failover, [], []): monitoring <20150.3134.0> from ' ns_1@10.2.1.61 ' [error_logger:info,2013-02-02T1:58:36.756, ns_1@10.2.1.61 :error_logger<0.6.0>:ale_error_logger_handler:log_report:72] ** In diags node 63 (old orchestrator) I saw a lot vbucket_mover crashed around rebalance failed ========================CRASH REPORT=========================   crasher:     initial call: ns_single_vbucket_mover:mover/6     pid: <0.15041.759>     registered_name: []     exception exit: {unexpected_exit,{'EXIT',<0.7018.476>,shutdown}}       in function ns_single_vbucket_mover:spawn_and_wait/1       in call from ns_single_vbucket_mover:mover_inner/6       in call from misc:try_with_maybe_ignorant_after/2       in call from ns_single_vbucket_mover:mover/6     ancestors: [<0.7018.476>,<0.5453.476>]     messages: [{'EXIT',<0.7018.476>,shutdown}]     links: [<0.7018.476>]     dictionary: [{cleanup_list,[<0.15227.759>,<0.16464.759>]}]     trap_exit: true     status: running     heap_size: 2584     stack_size: 24     reductions: 5950   neighbours: ** Time stamp of this test: Bucket "sasl" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.63 20:17:01 - Fri Feb 1, 2013 Started rebalancing bucket sasl ns_rebalancer000 ns_1@10.2.1.63 20:16:58 - Fri Feb 1, 2013 Starting rebalance, KeepNodes = [' ns_1@10.2.1.64 ',' ns_1@10.2.1.63 ', ' ns_1@10.2.1.61 '], EjectNodes = [] ns_orchestrator004 ns_1@10.2.1.63 20:16:58 - Fri Feb 1, 2013 Rebalance failed at this phase RB-1 right before complete rebalance of first bucket. Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.2.1.61 01:58:40 - Sat Feb 2, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 01:58:36 - Sat Feb 2, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 02:12:25 - Sat Feb 2, 2013 Link to manifest file http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.1-144-rel.setup.exe.manifest.xml Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201301/3nodes-phy-servers-201-144-reb-stopped-by-janitor-20130201-183832.tgz
Hide
Permalink
Aleksey Kondratenko added a comment -
I guess that's "it" ? Physical boxes right ?
Show
Aleksey Kondratenko added a comment - I guess that's "it" ? Physical boxes right ?
Hide
Permalink
Thuan Nguyen added a comment -
Yes, we dedicate 4 physical servers to run this test, not virtual servers.
Show
Thuan Nguyen added a comment - Yes, we dedicate 4 physical servers to run this test, not virtual servers.
Hide
Permalink
Farshid Ghods added a comment -
Assigning this to Siri as he was working with Tony for running this test and waiting for diags.
Please feel free to assign back to alk
Show
Farshid Ghods added a comment - Assigning this to Siri as he was working with Tony for running this test and waiting for diags. Please feel free to assign back to alk
Hide
Permalink
Aleksey Kondratenko added a comment -
We have diags.

No signs of our "usual" timeouts. But apparently some issue either timing or with timely delivering messages between hosts. Almost like network is badly undersized.
Show
Aleksey Kondratenko added a comment - We have diags. No signs of our "usual" timeouts. But apparently some issue either timing or with timely delivering messages between hosts. Almost like network is badly undersized.
Hide
Permalink
Sriram Melkote added a comment -
Network traces attached show no perceptible network stress or loss. All machines were reachable from all other machines throughout and no TCP disconnects. Need to investigate the possibility of disksup/memsup introduced failures as the next step.
Show
Sriram Melkote added a comment - Network traces attached show no perceptible network stress or loss. All machines were reachable from all other machines throughout and no TCP disconnects. Need to investigate the possibility of disksup/memsup introduced failures as the next step.
Hide
Permalink
Thuan Nguyen added a comment -
Continue rebalance cluster with network monitor script from Siri, rebalance failed again in few hours running with error

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.4548.4>,
{{wait_backfill_complete_failed_for,
[{'ns_1@10.2.1.61',
{'EXIT',
{normal,
{gen_server,call,
[<0.4516.4>,wait_backfill_complete,
infinity]}}}}]},
[{ns_single_vbucket_mover,
'-mover_inner/6-fun-3-',5}]}}}
ns_orchestrator002 ns_1@10.2.1.61 08:01:39 - Tue Feb 5, 2013
<0.27433.3> exited with {unexpected_exit,
{'EXIT',<0.4548.4>,
{{wait_backfill_complete_failed_for,
[{'ns_1@10.2.1.61',
{'EXIT',
{normal,
{gen_server,call,
[<0.4516.4>,wait_backfill_complete,
infinity]}}}}]},
[{ns_single_vbucket_mover,'-mover_inner/6-fun-3-',
5}]}}} ns_vbucket_mover000 ns_1@10.2.1.61 08:01:39 - Tue Feb 5, 2013


Link to collect info of all nodes in cluster of this failure
https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201302/3nodes-phy-servers-201-144-reb-failed-wait-backfill-20130205-003508.tgz
Show
Thuan Nguyen added a comment - Continue rebalance cluster with network monitor script from Siri, rebalance failed again in few hours running with error Rebalance exited with reason {unexpected_exit, {'EXIT',<0.4548.4>, {{wait_backfill_complete_failed_for, [{' ns_1@10.2.1.61 ', {'EXIT', {normal, {gen_server,call, [<0.4516.4>,wait_backfill_complete, infinity]}}}}]}, [{ns_single_vbucket_mover, '-mover_inner/6-fun-3-',5}]}}} ns_orchestrator002 ns_1@10.2.1.61 08:01:39 - Tue Feb 5, 2013 <0.27433.3> exited with {unexpected_exit, {'EXIT',<0.4548.4>, {{wait_backfill_complete_failed_for, [{' ns_1@10.2.1.61 ', {'EXIT', {normal, {gen_server,call, [<0.4516.4>,wait_backfill_complete, infinity]}}}}]}, [{ns_single_vbucket_mover,'-mover_inner/6-fun-3-', 5}]}}} ns_vbucket_mover000 ns_1@10.2.1.61 08:01:39 - Tue Feb 5, 2013 Link to collect info of all nodes in cluster of this failure https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201302/3nodes-phy-servers-201-144-reb-failed-wait-backfill-20130205-003508.tgz
Hide
Permalink
Jin Lim added a comment -
MB-6595 also have seen simliar issue, please take a look the latest finding from ns server team. For instance:

Aliaksey figured out that our tuning option that helped to decrease minor page faults on GNU/Linux does not apply to windows as apparently they are not using equivalent of mmap to allocate memory on windows.
Show
Jin Lim added a comment - MB-6595 also have seen simliar issue, please take a look the latest finding from ns server team. For instance: Aliaksey figured out that our tuning option that helped to decrease minor page faults on GNU/Linux does not apply to windows as apparently they are not using equivalent of mmap to allocate memory on windows.
Hide
Permalink
Jin Lim added a comment -
Per bug scrubs engineering (QE+Dev) continue to test and investigate the issue. However, we will be targeting the fix for 2.0.2.
Show
Jin Lim added a comment - Per bug scrubs engineering (QE+Dev) continue to test and investigate the issue. However, we will be targeting the fix for 2.0.2.
Hide
Permalink
Thuan Nguyen added a comment -

Environment:
** 4 physical servers. Each server has 32GB RAM, 4 core CPU and 2 regular spin disks

10.2.1.61
10.2.1.62
10.2.1.63
10.2.1.64

** 3 nodes cluster with build 2.0.1-144, remove node 64 and rebalance

Look at log, we can see rebalance out one node failed right after done rebalance first bucket. Four hours later, orchestrator was shutdown by memcached



Shutting down bucket "sasl" on 'ns_1@10.2.1.61' for server shutdown ns_memcached002 ns_1@10.2.1.61 04:52:50 - Tue Feb 12, 2013
Shutting down bucket "default" on 'ns_1@10.2.1.61' for server shutdown ns_memcached002 ns_1@10.2.1.61 04:52:45 - Tue Feb 12, 2013



Node 'ns_1@10.2.1.63' saw that node 'ns_1@10.2.1.61' came up. ns_node_disco004 ns_1@10.2.1.63 04:52:07 - Tue Feb 12, 2013
Node 'ns_1@10.2.1.64' saw that node 'ns_1@10.2.1.61' came up. ns_node_disco004 ns_1@10.2.1.64 04:52:01 - Tue Feb 12, 2013
Node 'ns_1@10.2.1.61' saw that node 'ns_1@10.2.1.64' went down. ns_node_disco005 ns_1@10.2.1.61 04:52:00 - Tue Feb 12, 2013
Node 'ns_1@10.2.1.61' saw that node 'ns_1@10.2.1.63' went down. ns_node_disco005 ns_1@10.2.1.61 04:52:00 - Tue Feb 12, 2013
Node 'ns_1@10.2.1.63' saw that node 'ns_1@10.2.1.61' went down. ns_node_disco005 ns_1@10.2.1.63 04:51:33 - Tue Feb 12, 2013
Node 'ns_1@10.2.1.64' saw that node 'ns_1@10.2.1.61' went down. ns_node_disco005 ns_1@10.2.1.64 04:51:25 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. (repeated 1 times) mb_master000 ns_1@10.2.1.63 04:51:15 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:50:16 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 04:49:52 - Tue Feb 12, 2013



Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:48:38 - Tue Feb 12, 2013
menelaus_web019 ns_1@10.2.1.61 04:47:45 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:47:42 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. (repeated 2 times) mb_master000 ns_1@10.2.1.63 04:47:15 - Tue Feb 12, 2013



Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.64 04:46:32 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:46:08 - Tue Feb 12, 2013



Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.2.1.61 00:46:13 - Tue Feb 12, 2013
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 00:46:13 - Tue Feb 12, 2013
Somebody thinks we're master. Not forcing mastership takover over ourselves mb_master000 ns_1@10.2.1.61 00:46:01 - Tue Feb 12, 2013


Shutting down bucket "sasl" on 'ns_1@10.2.1.64' for deletion ns_memcached002 ns_1@10.2.1.64 00:34:16 - Tue Feb 12, 2013
Bucket "default" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.61 00:34:16 - Tue Feb 12, 2013
Started rebalancing bucket default ns_rebalancer000 ns_1@10.2.1.61 00:34:16 - Tue Feb 12, 2013
Bucket "sasl" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.61 16:34:31 - Mon Feb 11, 2013
Started rebalancing bucket sasl ns_rebalancer000 ns_1@10.2.1.61 16:34:30 - Mon Feb 11, 2013
Starting rebalance, KeepNodes = ['ns_1@10.2.1.63','ns_1@10.2.1.61'], EjectNodes = ['ns_1@10.2.1.64']
ns_orchestrator004 ns_1@10.2.1.61 16:34:30 - Mon Feb 11, 2013


Here is the collect info file of those nodes
https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201302/3nodes-phy-servers-201-144-resetting-reb-bt-bucket-20130212-101159.tgz
Show
Thuan Nguyen added a comment - Environment: ** 4 physical servers. Each server has 32GB RAM, 4 core CPU and 2 regular spin disks 10.2.1.61 10.2.1.62 10.2.1.63 10.2.1.64 ** 3 nodes cluster with build 2.0.1-144, remove node 64 and rebalance Look at log, we can see rebalance out one node failed right after done rebalance first bucket. Four hours later, orchestrator was shutdown by memcached Shutting down bucket "sasl" on ' ns_1@10.2.1.61 ' for server shutdown ns_memcached002 ns_1@10.2.1.61 04:52:50 - Tue Feb 12, 2013 Shutting down bucket "default" on ' ns_1@10.2.1.61 ' for server shutdown ns_memcached002 ns_1@10.2.1.61 04:52:45 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.63 ' saw that node ' ns_1@10.2.1.61 ' came up. ns_node_disco004 ns_1@10.2.1.63 04:52:07 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.64 ' saw that node ' ns_1@10.2.1.61 ' came up. ns_node_disco004 ns_1@10.2.1.64 04:52:01 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.61 ' saw that node ' ns_1@10.2.1.64 ' went down. ns_node_disco005 ns_1@10.2.1.61 04:52:00 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.61 ' saw that node ' ns_1@10.2.1.63 ' went down. ns_node_disco005 ns_1@10.2.1.61 04:52:00 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.63 ' saw that node ' ns_1@10.2.1.61 ' went down. ns_node_disco005 ns_1@10.2.1.63 04:51:33 - Tue Feb 12, 2013 Node ' ns_1@10.2.1.64 ' saw that node ' ns_1@10.2.1.61 ' went down. ns_node_disco005 ns_1@10.2.1.64 04:51:25 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. (repeated 1 times) mb_master000 ns_1@10.2.1.63 04:51:15 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:50:16 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 04:49:52 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:48:38 - Tue Feb 12, 2013 menelaus_web019 ns_1@10.2.1.61 04:47:45 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:47:42 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. (repeated 2 times) mb_master000 ns_1@10.2.1.63 04:47:15 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.64 04:46:32 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.63 04:46:08 - Tue Feb 12, 2013 Resetting rebalance status since it's not really running ns_janitor000 ns_1@10.2.1.61 00:46:13 - Tue Feb 12, 2013 Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.2.1.61 00:46:13 - Tue Feb 12, 2013 Somebody thinks we're master. Not forcing mastership takover over ourselves mb_master000 ns_1@10.2.1.61 00:46:01 - Tue Feb 12, 2013 Shutting down bucket "sasl" on ' ns_1@10.2.1.64 ' for deletion ns_memcached002 ns_1@10.2.1.64 00:34:16 - Tue Feb 12, 2013 Bucket "default" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.61 00:34:16 - Tue Feb 12, 2013 Started rebalancing bucket default ns_rebalancer000 ns_1@10.2.1.61 00:34:16 - Tue Feb 12, 2013 Bucket "sasl" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.2.1.61 16:34:31 - Mon Feb 11, 2013 Started rebalancing bucket sasl ns_rebalancer000 ns_1@10.2.1.61 16:34:30 - Mon Feb 11, 2013 Starting rebalance, KeepNodes = [' ns_1@10.2.1.63 ',' ns_1@10.2.1.61 '], EjectNodes = [' ns_1@10.2.1.64 '] ns_orchestrator004 ns_1@10.2.1.61 16:34:30 - Mon Feb 11, 2013 Here is the collect info file of those nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201302/3nodes-phy-servers-201-144-resetting-reb-bt-bucket-20130212-101159.tgz
Hide
Permalink
Sriram Melkote added a comment -
Tony, I've setup:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay to 30
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort to 60000

The machines need to be restarted. But I've not done so as I saw some load on the cluster.

Can you please reboot all machines, and try the rebalance again?
Show
Sriram Melkote added a comment - Tony, I've setup: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay to 30 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort to 60000 The machines need to be restarted. But I've not done so as I saw some load on the cluster. Can you please reboot all machines, and try the rebalance again?
Hide
Permalink
Thuan Nguyen added a comment -
Those phsical servers gave back to Pavel last week
Show
Thuan Nguyen added a comment - Those phsical servers gave back to Pavel last week
Hide
Permalink
Sriram Melkote added a comment - - edited
Hi Tony - the servers are back with us now and I've applied all the patches. It may be worthwhile to rerun the test with these patches. Thanks!
Show
Sriram Melkote added a comment - - edited Hi Tony - the servers are back with us now and I've applied all the patches. It may be worthwhile to rerun the test with these patches. Thanks!
Hide
Permalink
Farshid Ghods added a comment -
Tony,

did you rerun the test ?
Show
Farshid Ghods added a comment - Tony, did you rerun the test ?
Hide
Permalink
Thuan Nguyen added a comment -
Someone ran the test on these physical servers. I will verify it with Siri and Pavel
Show
Thuan Nguyen added a comment - Someone ran the test on these physical servers. I will verify it with Siri and Pavel
Hide
Permalink
Sriram Melkote added a comment -
It was me - I have finished now, please go ahead
Show
Sriram Melkote added a comment - It was me - I have finished now, please go ahead
Hide
Permalink
Sriram Melkote added a comment -
Ping - want to test some patches on an AWS cluster.
Show
Sriram Melkote added a comment - Ping - want to test some patches on an AWS cluster.
Hide
Permalink
Sriram Melkote added a comment -
This appears to be possibly a manifestation of MB-7920. I've verified that network connectivity is never lost, and the Erlang node connectivity is itself stable over long periods by themselves. Potentially, the trigger of cluster to disconnect is the bucket deletion.
Show
Sriram Melkote added a comment - This appears to be possibly a manifestation of MB-7920 . I've verified that network connectivity is never lost, and the Erlang node connectivity is itself stable over long periods by themselves. Potentially, the trigger of cluster to disconnect is the bucket deletion.
Hide
Permalink
Sriram Melkote added a comment - - edited
Shifting this investigation to MB-7902 which has identical problem, but is easier to reproduce. Normally, I'd prefer to keep the older bug open (this) and close the newer bug as duplicate, but making an exception in this case as the newer bug MB-7902 more narrowly shows this problem.
Show
Sriram Melkote added a comment - - edited Shifting this investigation to MB-7902 which has identical problem, but is easier to reproduce. Normally, I'd prefer to keep the older bug open (this) and close the newer bug as duplicate, but making an exception in this case as the newer bug MB-7902 more narrowly shows this problem.

People

  • Assignee:
    Thuan Nguyen
    Reporter:
    Farshid Ghods
Vote (0)
Watch (6)

Dates

  • Created:
    Updated:
    Resolved: