[MB-5434] rebalance fails with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt Created: 04/Jun/12  Updated: 09/Jan/13  Resolved: 05/Jun/12

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 1.7.2
Fix Version/s: 1.8.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Farshid Ghods (Inactive) Assignee: Aleksey Kondratenko
Resolution: Fixed Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2012-06-04 at 1.30.06 AM.png    

 Description   
obvious from these logs :
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
(repeated 1 times) ns_orchestrator002 ns_1@10.113.11.11 01:24:57 - Mon Jun 4, 2012
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
ns_orchestrator002 ns_1@10.113.11.11 01:19:18 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
ns_orchestrator004 ns_1@10.113.11.11 01:19:07 - Mon Jun 4, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.113.11.11 01:13:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.17' for server shutdown ns_memcached002 ns_1@10.113.11.17 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.19' for server shutdown ns_memcached002 ns_1@10.113.11.19 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.23' for server shutdown ns_memcached002 ns_1@10.113.11.23 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.11' for server shutdown ns_memcached002 ns_1@10.113.11.11 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.12' for server shutdown ns_memcached002 ns_1@10.113.11.12 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.5' for server shutdown ns_memcached002 ns_1@10.113.11.5 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.16' for server shutdown ns_memcached002 ns_1@10.113.11.16 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.24' for server shutdown ns_memcached002 ns_1@10.113.11.24 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.21' for server shutdown ns_memcached002 ns_1@10.113.11.21 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.27' for server shutdown ns_memcached002 ns_1@10.113.11.27 01:02:19 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']

 Comments   
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
Truncated log event:
{info_msg,<0.50.0>,
          {<0.29774.5803>,
           "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",
           ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101,
            "Authentication",
            [{num_replicas,2},
             {ram_quota,104857600},
             {auth_type,sasl},
             {sasl_password,"soWeefeot7ai"},
             {type,membase},
             {num_vbuckets,1024},
             {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',
                       'ns_1@10.113.11.3','ns_1@10.113.11.4',
                       'ns_1@10.113.11.6','ns_1@10.113.11.7',
                       'ns_1@10.113.11.8','ns_1@10.113.11.9',
                       'ns_1@10.113.11.10','ns_1@10.113.11.13',
                       'ns_1@10.113.11.14','ns_1@10.113.11.15',
                       'ns_1@10.113.11.18','ns_1@10.113.11.20',
                       'ns_1@10.113.11.22','ns_1@10.113.11.25',
                       'ns_1@10.113.11.26','ns_1@10.113.11.28',
                       'ns_1@10.113.11.29','ns_1@10.113.11.30']},
             {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.14',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.14',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.15',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.10',
                    'ns_1@10.113.11.29'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.10',
                    'ns_1@10.113.11.29'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.13',
                    'ns_1@10.113.11.28'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.29',
                    'ns_1@10.113.11.9'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.26','ns_1@10.113.11.7',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.26','ns_1@10.113.11.8',
                    'ns_1@10.113.11.15'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.29',
                    'ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.3',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.3',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.18',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.1',
                    'ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.1',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.18',
                    'ns_1@10.113.11.2'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.7','ns_1@10.113.11.13',
                    'ns_1@10.113.11.30'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.14',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.15',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.20',
                    'ns_1@10.113.11.26'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.20',
                    'ns_1@10.113.11.26'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.22',
                    'ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.25',
                    'ns_1@10.113.11.4'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...],
                   ['ns_1@10.113.11.15'|...],
                   [...]|...]},
             {fastForwardMap,undefined}]]}}
INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

config change:
rebalance_status ->
running

INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

ns_node_disco_conf_events config all

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config done

ERROR REPORT <5881.6.0> 2012-06-04 04:19:07
===============================================================================

Truncated log event:
{info_msg,<0.50.0>,
          {<0.164.0>,"config change:~n~p ->~n~s~n",
           [buckets,
            <<"[{configs,[{\"Authentication\",\n [{num_replicas,2},\n {ram_quota,104857600},\n {auth_type,sasl},\n {sasl_password,\"soWeefeot7ai\"},\n {type,membase},\n {num_vbuckets,1024},\n {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>]}}
INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

ns_node_disco_conf_events config all

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config done

ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07
===============================================================================

ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication":
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'stats_reader-Authentication','ns_1@10.113.11.11'},
             {latest,minute}]}}}


INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
Truncated log event:
{info_msg,<0.50.0>,
          {<0.29774.5803>,
           "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",
           ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101,
            "Authentication",
            [{num_replicas,2},
             {ram_quota,104857600},
             {auth_type,sasl},
             {sasl_password,"soWeefeot7ai"},
             {type,membase},
             {num_vbuckets,1024},
             {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',
                       'ns_1@10.113.11.3','ns_1@10.113.11.4',
                       'ns_1@10.113.11.6','ns_1@10.113.11.7',
                       'ns_1@10.113.11.8','ns_1@10.113.11.9',
                       'ns_1@10.113.11.10','ns_1@10.113.11.13',
                       'ns_1@10.113.11.14','ns_1@10.113.11.15',
                       'ns_1@10.113.11.18','ns_1@10.113.11.20',
                       'ns_1@10.113.11.22','ns_1@10.113.11.25',
                       'ns_1@10.113.11.26','ns_1@10.113.11.28',
                       'ns_1@10.113.11.29','ns_1@10.113.11.30']},
             {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.10','ns_1@10.113.11.20',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.14',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.14',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.15',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.10',
                    'ns_1@10.113.11.29'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.10',
                    'ns_1@10.113.11.29'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.13',
                    'ns_1@10.113.11.28'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.29',
                    'ns_1@10.113.11.9'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.26','ns_1@10.113.11.7',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.26','ns_1@10.113.11.8',
                    'ns_1@10.113.11.15'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.29',
                    'ns_1@10.113.11.7'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.3',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.3',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.6',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.18',
                    'ns_1@10.113.11.25'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.14','ns_1@10.113.11.1',
                    'ns_1@10.113.11.10'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.29','ns_1@10.113.11.9',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.18','ns_1@10.113.11.1',
                    'ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.4',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.22','ns_1@10.113.11.1',
                    'ns_1@10.113.11.14'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.28','ns_1@10.113.11.9',
                    'ns_1@10.113.11.22'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.2','ns_1@10.113.11.10',
                    'ns_1@10.113.11.18'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.18',
                    'ns_1@10.113.11.2'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'],
                   ['ns_1@10.113.11.7','ns_1@10.113.11.13',
                    'ns_1@10.113.11.30'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.14',
                    'ns_1@10.113.11.1'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.15',
                    'ns_1@10.113.11.20'],
                   ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.20',
                    'ns_1@10.113.11.26'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.20',
                    'ns_1@10.113.11.26'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.22',
                    'ns_1@10.113.11.3'],
                   ['ns_1@10.113.11.13','ns_1@10.113.11.25',
                    'ns_1@10.113.11.4'],
                   ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...],
                   ['ns_1@10.113.11.15'|...],
                   [...]|...]},
             {fastForwardMap,undefined}]]}}
INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

config change:
rebalance_status ->
running

INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

ns_node_disco_conf_events config all

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config done

ERROR REPORT <5881.6.0> 2012-06-04 04:19:07
===============================================================================

Truncated log event:
{info_msg,<0.50.0>,
          {<0.164.0>,"config change:~n~p ->~n~s~n",
           [buckets,
            <<"[{configs,[{\"Authentication\",\n [{num_replicas,2},\n {ram_quota,104857600},\n {auth_type,sasl},\n {sasl_password,\"soWeefeot7ai\"},\n {type,membase},\n {num_vbuckets,1024},\n {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>]}}
INFO REPORT <5881.164.0> 2012-06-04 04:19:07
===============================================================================

ns_node_disco_conf_events config all

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config

INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
===============================================================================

Pushing config done

ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07
===============================================================================

ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication":
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'stats_reader-Authentication','ns_1@10.113.11.11'},
             {latest,minute}]}}}


INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
Indeed we set set of bucket's list nodes to current list of nodes even if some of them are going to be rebalanced out. That's indeed going to start buckets on those previously nearly rebalanced out nodes. I think we can just reverse list of buckets before rebalancing as a workaround.

This is the /diag/eval snipped I used to revers list of buckets:

ns_config:update_key(buckets, fun ([{configs, Buckets0}]) -> Buckets = lists:reverse(Buckets0), [{configs, Buckets}] end).
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
>>>ns_config:update_key(buckets, fun ([{configs, Buckets0}]) -> Buckets = lists:reverse(Buckets0), [{configs, Buckets}] end).

what does bucket0 mean here ? should we pass the name of the bucket that is already rebalanced out ?
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
does it reverse by the alphabetical order or it reverses based on the state of the bucket ( is it rebalanced out or not yet )
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
it reverses current list. Buckets0 is current list.
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
I think we can do better than reversing. Give me few more minutes
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
This snipped rotates list of buckets instead:

ns_config:update_key(buckets, fun ([{configs, Buckets}]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [{configs, ShuffledBuckets}] end).
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
the problem with reversing is that if i have 5 buckets
a,b,c,d,e,f
first rebalance succeeds and remove "a"
then rebalance is stuck and we restart rebalancing but fails with wait_for_memcached ("a")

so reverse bucket list f,e,d,c,b,a

rebalance moves f and then is stuck while moving e

now reversing won't help anymore because both vbucket "f" and "a" are rebalanced out and the rest are still imbalanced
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
exactly, So rotating would change a,b,c,d,e,f to b,c,d,e,f,a
Comment by Farshid Ghods (Inactive) [ 04/Jun/12 ]
>>ns_config:update_key(buckets, fun ([{configs, Buckets}]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [{configs, ShuffledBuckets}] end).

so this script moves the buckets that are still part of the cluster to the head of list and the "rebalanced_out" ones to the end ?
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
right.
Comment by Aleksey Kondratenko [ 04/Jun/12 ]
Right. The idea is in worst case you can rebalance bucket after bucket. And then usual failover would just get rid of this nodes being in server list of buckets
Comment by Aleksey Kondratenko [ 05/Jun/12 ]
fixed for 1.8.1: http://review.couchbase.org/16783
Comment by Thuan Nguyen [ 05/Jun/12 ]
Integrated in github-ns-server-2-0 #364 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/364/])
    MB-5434: don't add ejected nodes back to bucket's servers list (Revision 9ecbe6c6f1c596faf995dabc7e2abb3facd4c8b7)

     Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/ns_rebalancer.erl
Comment by Ketaki Gangal [ 21/Jun/12 ]
Tested on build 927 - Works fine. Seeing no error.
Generated at Tue Sep 23 08:51:50 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.