Couchbase Server

[windows] Rebalance exited with reason {detected_nodes_change, {ns_node_disco_events,

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Blocker Blocker
  • Resolution: Unresolved
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.2
  • Component/s: ns_server
  • Security Level: Public
  • Environment:
    build 2.0.1-179-rel

Description


The below test is failing while doing rebalance:
./testrunner -i win-aws.ini -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=2,delete-ratio=0.6,expiry-ratio=0.2,do-stop=True

Logs show below error message:

<0.6654.65>:ns_orchestrator:handle_info:319]Rebalance exited with reason {detected_nodes_change,
                                 {ns_node_disco_events,
                                     ['ns_1@10.130.71.207',
                                      'ns_1@10.131.33.89',
                                      'ns_1@10.131.35.139',
                                      'ns_1@10.142.174.97'],

The run log of testrunner is attached. Will be attaching cbcollect_info shortly.
  1. File
    dist_util.beam
    26/Apr/13 10:46 PM
    21 kB
    Sriram Melkote
  2. File
    nohup.out.multi-nodes-2.0.x-windows-64-rebalance-kv
    13/Mar/13 8:12 AM
    1.09 MB
    Deepkaran Salooja

Issue Links

depends on

Bug - A problem which impairs or prevents the functions of the product. MB-7658 [Windows] severe timeouts in windows cluster with 2 buckets and 1 ddoc per bucket causes rebalance and other failures

  • Blocker - Blocks development and/or testing work, production could not run.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Bug - A problem which impairs or prevents the functions of the product. MB-7920 [windows] Rebalancing out the last node causes the cluster to be reinitialized

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.
duplicates

Bug - A problem which impairs or prevents the functions of the product. MB-7658 [Windows] severe timeouts in windows cluster with 2 buckets and 1 ddoc per bucket causes rebalance and other failures

  • Blocker - Blocks development and/or testing work, production could not run.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Bug - A problem which impairs or prevents the functions of the product. MB-7949 [windows] Rebalance exited with reason wait_checkpoint_persisted_failed due to nodedown {error,closed}

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.
is duplicated by

Bug - A problem which impairs or prevents the functions of the product. MB-7891 [windows] a node became down during rebalance

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.

Bug - A problem which impairs or prevents the functions of the product. MB-7946 [windows] Rebalance exited with reason {{bulk_set_vbucket_state_failed, {{badmatch,wrong_rebalancer_pid}

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Deepkaran Salooja added a comment - - edited
collect info:

https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.142.174.97.zip
https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.33.89.zip
https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.35.139.zip
https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.71.207.zip
Show
Deepkaran Salooja added a comment - - edited collect info: https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.142.174.97.zip https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.33.89.zip https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.35.139.zip https://s3.amazonaws.com/bugdb/jira/MB-7902/e9125b6b/cbcollect_info_10.131.71.207.zip
Hide
Permalink
Thuan Nguyen added a comment -
I got the same rebalance failure in physical servers during rebalance out. I will add collect info file of all nodes soon
Show
Thuan Nguyen added a comment - I got the same rebalance failure in physical servers during rebalance out. I will add collect info file of all nodes soon
Hide
Permalink
Thuan Nguyen added a comment -
collect info of all nodes running build 2.0.1-175
rebalance failed during rebalance out node 63
https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201303/4win-nodes-201_175-reb-out-node-change-20130314-111111.tgz
Show
Thuan Nguyen added a comment - collect info of all nodes running build 2.0.1-175 rebalance failed during rebalance out node 63 https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201303/4win-nodes-201_175-reb-out-node-change-20130314-111111.tgz
Hide
Permalink
Sriram Melkote added a comment -
Potentially the same issue as MB-7658
Show
Sriram Melkote added a comment - Potentially the same issue as MB-7658
Hide
Permalink
Sriram Melkote added a comment -
I too ran this many times but couldn't reproduce. But the bug is real, I've seen it happen here and elsewhere.
Show
Sriram Melkote added a comment - I too ran this many times but couldn't reproduce. But the bug is real, I've seen it happen here and elsewhere.
Hide
Permalink
Ketaki Gangal added a comment -
Does rebalance eventually succeed on this test?
Show
Ketaki Gangal added a comment - Does rebalance eventually succeed on this test?
Hide
Permalink
Sriram Melkote added a comment -
Yes - we can restart the rebalance and it will complete. It's not reproducible by running the specific test, it's bit random failure.
Show
Sriram Melkote added a comment - Yes - we can restart the rebalance and it will complete. It's not reproducible by running the specific test, it's bit random failure.
Hide
Permalink
Sriram Melkote added a comment -
This appears to be possibly a manifestation of MB-7920. I've verified that network connectivity is never lost, and the Erlang node connectivity is itself stable over long periods by themselves. Potentially, the trigger of cluster to disconnect is the bucket deletion.
Show
Sriram Melkote added a comment - This appears to be possibly a manifestation of MB-7920 . I've verified that network connectivity is never lost, and the Erlang node connectivity is itself stable over long periods by themselves. Potentially, the trigger of cluster to disconnect is the bucket deletion.
Hide
Permalink
Sriram Melkote added a comment -
MB7920 does not reproduce despite many tests, so investigating this independently.
Show
Sriram Melkote added a comment - MB7920 does not reproduce despite many tests, so investigating this independently.
Hide
Permalink
Thuan Nguyen added a comment - - edited
Hit this bug again in build 2.0.1-185. Node down during rebalance due to net_tick_timeout.
Environment:
Create a cluster with 3 nodes
10.2.1.61
10.2.1.62
10.2.1.63
Create 2 buckets: default (14GB) and sasl (10GB)
No view or xdcr created
Load 20+ million items to both bucket until resident ratio on both bucket around 90%
Access cluster in 3 hours with spec in this page http://hub.internal.couchbase.com/confluence/pages/viewpage.action?pageId=6785119
Add node 10.2.1.64 to cluster and rebalance. Rebalance failed MB-7995
Then rebalance again, rebalance hang MB-7996
Then rebalance again. After failed several times with errors in bug MB-7658, MB-7902, rebalance success
Remove node 63. After one rebalance failed (node change error), rebalance completed
Swap rebalance (add node 63 back and remove node 61), rebalance failed due to node 62 and 63 down

Link to collect info files of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201304/4phy-win-201_185-reb-failed_node-down-net_tick_timeout_20130408-125052.tgz

Link to erl dump https://s3.amazonaws.com/packages.couchbase/erlang/windows/erl_node_63.DMP

Link to erl dump zip file https://s3.amazonaws.com/packages.couchbase/erlang/windows/erl_node_63.DMP.zip
Show
Thuan Nguyen added a comment - - edited Hit this bug again in build 2.0.1-185. Node down during rebalance due to net_tick_timeout. Environment: Create a cluster with 3 nodes 10.2.1.61 10.2.1.62 10.2.1.63 Create 2 buckets: default (14GB) and sasl (10GB) No view or xdcr created Load 20+ million items to both bucket until resident ratio on both bucket around 90% Access cluster in 3 hours with spec in this page http://hub.internal.couchbase.com/confluence/pages/viewpage.action?pageId=6785119 Add node 10.2.1.64 to cluster and rebalance. Rebalance failed MB-7995 Then rebalance again, rebalance hang MB-7996 Then rebalance again. After failed several times with errors in bug MB-7658 , MB-7902 , rebalance success Remove node 63. After one rebalance failed (node change error), rebalance completed Swap rebalance (add node 63 back and remove node 61), rebalance failed due to node 62 and 63 down Link to collect info files of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201304/4phy-win-201_185-reb-failed_node-down-net_tick_timeout_20130408-125052.tgz Link to erl dump https://s3.amazonaws.com/packages.couchbase/erlang/windows/erl_node_63.DMP Link to erl dump zip file https://s3.amazonaws.com/packages.couchbase/erlang/windows/erl_node_63.DMP.zip
Hide
Permalink
Thuan Nguyen added a comment -
Promote it to blocker since I see it often when testing in windows physical machines
Show
Thuan Nguyen added a comment - Promote it to blocker since I see it often when testing in windows physical machines
Hide
Permalink
Sriram Melkote added a comment -
I've attached dist_util.beam that will retry on net_tick_timeout. Can you please replace C:\Program Files\Couchbase\Server\lib\kernel-2.14.5\ebin\dist_util.beam with the attached version, and rerun the test and see if it does anything to the issue.
Show
Sriram Melkote added a comment - I've attached dist_util.beam that will retry on net_tick_timeout. Can you please replace C:\Program Files\Couchbase\Server\lib\kernel-2.14.5\ebin\dist_util.beam with the attached version, and rerun the test and see if it does anything to the issue.
Hide
Permalink
Maria McDuff added a comment -
siri still debugging this issue. probl still reproducible.
Show
Maria McDuff added a comment - siri still debugging this issue. probl still reproducible.
Hide
Permalink
Ray Chin added a comment -
The 4 machines have been moved to another desk near alk.
Show
Ray Chin added a comment - The 4 machines have been moved to another desk near alk.

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Deepkaran Salooja
Vote (0)
Watch (8)

Dates

  • Created:
    Updated: