rebalance fails with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt

Ketaki Gangal added a comment - 21/Jun/12 7:12 PM Tested on build 927 - Works fine. Seeing no error.

Thuan Nguyen added a comment - 05/Jun/12 10:49 PM Integrated in github-ns-server-2-0 #364 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/364/ ])      MB-5434 : don't add ejected nodes back to bucket's servers list (Revision 9ecbe6c6f1c596faf995dabc7e2abb3facd4c8b7)      Result = SUCCESS Aliaksey Kandratsenka : Files : * src/ns_rebalancer.erl

Aleksey Kondratenko added a comment - 05/Jun/12 3:42 PM fixed for 1.8.1: http://review.couchbase.org/16783

Aleksey Kondratenko added a comment - 04/Jun/12 4:04 PM Right. The idea is in worst case you can rebalance bucket after bucket. And then usual failover would just get rid of this nodes being in server list of buckets

Aleksey Kondratenko added a comment - 04/Jun/12 4:02 PM right.

Farshid Ghods added a comment - 04/Jun/12 4:00 PM >>ns_config:update_key(buckets, fun ([{configs, Buckets}]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [{configs, ShuffledBuckets}] end). so this script moves the buckets that are still part of the cluster to the head of list and the "rebalanced_out" ones to the end ?

Aleksey Kondratenko added a comment - 04/Jun/12 3:52 PM exactly, So rotating would change a,b,c,d,e,f to b,c,d,e,f,a

Farshid Ghods added a comment - 04/Jun/12 3:31 PM the problem with reversing is that if i have 5 buckets a,b,c,d,e,f first rebalance succeeds and remove "a" then rebalance is stuck and we restart rebalancing but fails with wait_for_memcached ("a") so reverse bucket list f,e,d,c,b,a rebalance moves f and then is stuck while moving e now reversing won't help anymore because both vbucket "f" and "a" are rebalanced out and the rest are still imbalanced

Aleksey Kondratenko added a comment - 04/Jun/12 3:30 PM This snipped rotates list of buckets instead: ns_config:update_key(buckets, fun ([{configs, Buckets}]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [{configs, ShuffledBuckets}] end).

Aleksey Kondratenko added a comment - 04/Jun/12 3:24 PM I think we can do better than reversing. Give me few more minutes

Aleksey Kondratenko added a comment - 04/Jun/12 3:24 PM it reverses current list. Buckets0 is current list.

Farshid Ghods added a comment - 04/Jun/12 2:55 PM does it reverse by the alphabetical order or it reverses based on the state of the bucket ( is it rebalanced out or not yet )

Farshid Ghods added a comment - 04/Jun/12 2:17 PM >>>ns_config:update_key(buckets, fun ([{configs, Buckets0}]) -> Buckets = lists:reverse(Buckets0), [{configs, Buckets}] end). what does bucket0 mean here ? should we pass the name of the bucket that is already rebalanced out ?

Aleksey Kondratenko added a comment - 04/Jun/12 1:56 PM Indeed we set set of bucket's list nodes to current list of nodes even if some of them are going to be rebalanced out. That's indeed going to start buckets on those previously nearly rebalanced out nodes. I think we can just reverse list of buckets before rebalancing as a workaround. This is the /diag/eval snipped I used to revers list of buckets: ns_config:update_key(buckets, fun ([{configs, Buckets0}]) -> Buckets = lists:reverse(Buckets0), [{configs, Buckets}] end).

Farshid Ghods added a comment - 04/Jun/12 12:54 PM Truncated log event: {info_msg,<0.50.0>,           {<0.29774.5803>,            "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",            [' ns_1@10.113.11.11 ',<0.29774.5803>,ns_rebalancer,101,             "Authentication",             [{num_replicas,2},              {ram_quota,104857600},              {auth_type,sasl},              {sasl_password,"soWeefeot7ai"},              {type,membase},              {num_vbuckets,1024},              {servers,[' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',                        ' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',                        ' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',                        ' ns_1@10.113.11.8 ',' ns_1@10.113.11.9 ',                        ' ns_1@10.113.11.10 ',' ns_1@10.113.11.13 ',                        ' ns_1@10.113.11.14 ',' ns_1@10.113.11.15 ',                        ' ns_1@10.113.11.18 ',' ns_1@10.113.11.20 ',                        ' ns_1@10.113.11.22 ',' ns_1@10.113.11.25 ',                        ' ns_1@10.113.11.26 ',' ns_1@10.113.11.28 ',                        ' ns_1@10.113.11.29 ',' ns_1@10.113.11.30 ']},              {map,[[' ns_1@10.113.11.10 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.15 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.29 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.29 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.13 ',                     ' ns_1@10.113.11.28 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.29 ',                     ' ns_1@10.113.11.9 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.26 ',' ns_1@10.113.11.7 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.26 ',' ns_1@10.113.11.8 ',                     ' ns_1@10.113.11.15 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.29 ',                     ' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.3 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.3 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.9 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.2 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.7 ',' ns_1@10.113.11.13 ',                     ' ns_1@10.113.11.30 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.15 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.26 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.26 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.22 ',                     ' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.25 ',                     ' ns_1@10.113.11.4 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.28 '|...],                    [' ns_1@10.113.11.15 '|...],                    [...]|...]},              {fastForwardMap,undefined}]]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== config change: rebalance_status -> running INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.6.0> 2012-06-04 04:19:07 =============================================================================== Truncated log event: {info_msg,<0.50.0>,           {<0.164.0>,"config change:~n~p ->~n~s~n",            [buckets,             <<"[{configs,[{\"Authentication\",\n [{num_replicas,2},\n {ram_quota,104857600},\n {auth_type,sasl},\n {sasl_password,\"soWeefeot7ai\"},\n {type,membase},\n {num_vbuckets,1024},\n {servers,[' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',\n ' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',\n "...>>]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07 =============================================================================== ns_1@10.113.11.11 :<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication": {'EXIT',     {noproc,         {gen_server,call,             [{'stats_reader-Authentication',' ns_1@10.113.11.11 '},              {latest,minute}]}}} INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08

Farshid Ghods added a comment - 04/Jun/12 12:54 PM Truncated log event: {info_msg,<0.50.0>,           {<0.29774.5803>,            "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",            [' ns_1@10.113.11.11 ',<0.29774.5803>,ns_rebalancer,101,             "Authentication",             [{num_replicas,2},              {ram_quota,104857600},              {auth_type,sasl},              {sasl_password,"soWeefeot7ai"},              {type,membase},              {num_vbuckets,1024},              {servers,[' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',                        ' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',                        ' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',                        ' ns_1@10.113.11.8 ',' ns_1@10.113.11.9 ',                        ' ns_1@10.113.11.10 ',' ns_1@10.113.11.13 ',                        ' ns_1@10.113.11.14 ',' ns_1@10.113.11.15 ',                        ' ns_1@10.113.11.18 ',' ns_1@10.113.11.20 ',                        ' ns_1@10.113.11.22 ',' ns_1@10.113.11.25 ',                        ' ns_1@10.113.11.26 ',' ns_1@10.113.11.28 ',                        ' ns_1@10.113.11.29 ',' ns_1@10.113.11.30 ']},              {map,[[' ns_1@10.113.11.10 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.10 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.15 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.29 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.29 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.13 ',                     ' ns_1@10.113.11.28 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.29 ',                     ' ns_1@10.113.11.9 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.26 ',' ns_1@10.113.11.7 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.26 ',' ns_1@10.113.11.8 ',                     ' ns_1@10.113.11.15 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.29 ',                     ' ns_1@10.113.11.7 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.3 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.3 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.6 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.1 ',' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.25 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.9 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.14 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.10 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.29 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.18 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.4 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.3 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.22 ',' ns_1@10.113.11.1 ',                     ' ns_1@10.113.11.14 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.28 ',' ns_1@10.113.11.9 ',                     ' ns_1@10.113.11.22 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.4 ',' ns_1@10.113.11.8 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.2 ',' ns_1@10.113.11.10 ',                     ' ns_1@10.113.11.18 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.18 ',                     ' ns_1@10.113.11.2 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.6 ',' ns_1@10.113.11.7 ',' ns_1@10.113.11.8 '],                    [' ns_1@10.113.11.7 ',' ns_1@10.113.11.13 ',                     ' ns_1@10.113.11.30 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.14 ',                     ' ns_1@10.113.11.1 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.15 ',                     ' ns_1@10.113.11.20 '],                    [' ns_1@10.113.11.1 ',' ns_1@10.113.11.9 ',' ns_1@10.113.11.13 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.26 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.20 ',                     ' ns_1@10.113.11.26 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.22 ',                     ' ns_1@10.113.11.3 '],                    [' ns_1@10.113.11.13 ',' ns_1@10.113.11.25 ',                     ' ns_1@10.113.11.4 '],                    [' ns_1@10.113.11.15 ',' ns_1@10.113.11.28 '|...],                    [' ns_1@10.113.11.15 '|...],                    [...]|...]},              {fastForwardMap,undefined}]]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== config change: rebalance_status -> running INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.6.0> 2012-06-04 04:19:07 =============================================================================== Truncated log event: {info_msg,<0.50.0>,           {<0.164.0>,"config change:~n~p ->~n~s~n",            [buckets,             <<"[{configs,[{\"Authentication\",\n [{num_replicas,2},\n {ram_quota,104857600},\n {auth_type,sasl},\n {sasl_password,\"soWeefeot7ai\"},\n {type,membase},\n {num_vbuckets,1024},\n {servers,[' ns_1@10.113.11.1 ',' ns_1@10.113.11.2 ',\n ' ns_1@10.113.11.3 ',' ns_1@10.113.11.4 ',\n "...>>]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07 =============================================================================== ns_1@10.113.11.11 :<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication": {'EXIT',     {noproc,         {gen_server,call,             [{'stats_reader-Authentication',' ns_1@10.113.11.11 '},              {latest,minute}]}}} INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08