Details
Description
Have not seen this issue previously ever on 181. Seems to be something new.
Test case that failed
swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=3,num-buckets=5,num-swap=2,swap-orchestrator=True,keys-count=800000
1) Failover 2 nodes, including the orchestrator
2) Rebalance them out
3) Add them back and rebalance in, without cleaning them.
[rebalance:error] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27930.16>:ns_rebalancer:do_wait_buckets_shutdown:170] Failed to wait deletion of some buckets on some nodes: [{'ns_1@10.1.3.77',
{'EXIT',
{old_buckets_shutdown_wait_failed,
["bucket-2",
"bucket-1",
"bucket-0",
"bucket-4",
"bucket-3"]}}}]
[user:info] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27893.16>:ns_orchestrator:handle_info:246] Rebalance exited with reason {buckets_shutdown_wait_failed,
[{'ns_1@10.1.3.77',
{'EXIT',
{old_buckets_shutdown_wait_failed,
["bucket-2","bucket-1","bucket-0",
"bucket-4","bucket-3"]}}}]}
Test case that failed
swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=3,num-buckets=5,num-swap=2,swap-orchestrator=True,keys-count=800000
1) Failover 2 nodes, including the orchestrator
2) Rebalance them out
3) Add them back and rebalance in, without cleaning them.
[rebalance:error] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27930.16>:ns_rebalancer:do_wait_buckets_shutdown:170] Failed to wait deletion of some buckets on some nodes: [{'ns_1@10.1.3.77',
{'EXIT',
{old_buckets_shutdown_wait_failed,
["bucket-2",
"bucket-1",
"bucket-0",
"bucket-4",
"bucket-3"]}}}]
[user:info] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27893.16>:ns_orchestrator:handle_info:246] Rebalance exited with reason {buckets_shutdown_wait_failed,
[{'ns_1@10.1.3.77',
{'EXIT',
{old_buckets_shutdown_wait_failed,
["bucket-2","bucket-1","bucket-0",
"bucket-4","bucket-3"]}}}]}
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Karan Kumar
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Fix Version/s | 1.8.1 [ 10295 ] | |
| Affects Version/s | 1.8.1-release-candidate [ 10299 ] | |
| Component/s | ns_server [ 10019 ] |
Aleksey Kondratenko
made changes -
| Assignee | Aleksey Kondratenko [ alkondratenko ] | Frank Weigel [ frank ] |
Dipti Borkar
made changes -
| Labels | customer | |
| Assignee | Frank Weigel [ frank ] | Aleksey Kondratenko [ alkondratenko ] |
Aleksey Kondratenko
made changes -
| Assignee | Aleksey Kondratenko [ alkondratenko ] | Dipti Borkar [ dipti ] |
Farshid Ghods
made changes -
| Summary | Rebalance failed due to buckets_shutdown_wait_failed | user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase |
Dipti Borkar
made changes -
| Assignee | Dipti Borkar [ dipti ] | Chiyoung Seo [ chiyoung ] |
Dipti Borkar
made changes -
| Sprint Status | Current Sprint |
Dipti Borkar
made changes -
| Sprint Priority | 0 |
Aleksey Kondratenko
made changes -
| Assignee | Chiyoung Seo [ chiyoung ] | Aleksey Kondratenko [ alkondratenko ] |
Aleksey Kondratenko
made changes -
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Resolution | Fixed [ 1 ] |
Dipti Borkar
made changes -
| Sprint Status | Current Sprint | |
| Sprint Priority | 0 |
Farshid Ghods
made changes -
| Status | Resolved [ 5 ] | Closed [ 6 ] |