[MB-5625] Rebalance can fail with various error if user changes bucket password before rebalance process

Wed, 9 Jan 2013 22:59:34 -0600

Setup:
1.Setup a 18 node cluster. Enable Auto-failover
2.Load data on all 3 buckets [around 50M, 22M, 500k] items.
3. Continue loading data..
4. Remove orchestrator node [105] Add new node [126]
5. Issue rebalance on this cluster. Rebalance failed with "replicator died" - filed bug 5343
6. Re-issue rebalance . Rebalance fails with " change_filter_failed"

Output:
Reissue rebalance fails with "change_filter_failed"
Node 105 is auto-failed over

Rebalance exited with reason {{change_filter_failed,
{'EXIT',
{{badmatch,
{failed,
{error,
{badmatch,
{memcached_error,auth_error,
<<"Auth failure">>}},
[{ebucketmigrator_srv,connect,4},
{ebucketmigrator_srv,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}},
[{ns_vbm_sup,
'-local_change_vbucket_filter/4-fun-2-',6},
{misc,'-executing_on_new_process/1-fun-0-',
3}]}}},
[{ns_vbm_sup,change_vbucket_filter,4},
{ns_vbm_sup,'-set_replicas/3-fun-2-',5},
{lists,foldl,3},
{ns_vbm_sup,set_replicas,3},
{ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-',
3},
{lists,foreach,2},
{ns_vbm_sup,apply_changes,2},
{ns_vbucket_mover,sync_replicas,0}]}

Logs at - https://s3.amazonaws.com/bugdb/jira/bug4-rebalance-181/bug5.tar

Couchbase

[MB-5625] Rebalance can fail with various error if user changes bucket password before rebalance process