Rebalance exited with reason bulk_set_vbucket_state_failed exit timeout

Farshid Ghods added a comment - 17/Aug/12 3:43 PM i know this is a duplicate of some of the tests that Tony has been running but i also managed to reproduce this on a cluster with 11 ddocs and only 500k items in the cluster and 4 core cpu and 4 gb ram. the rebalance message says{update_vbucket_state,512,replica, undefined,' ns_1@10.1.3.70 '}}, 60000 which means there is a timeout but why the new node which is 70 does not have an active vbucket and i dont see any active task there also during rebalancing stats page was timing out which is expected in this case because rebalance failed eventually with timeout Aliaksey, can you please look at this cluster. I am afraid this is something very basic and we are losing track of issues because of the more advanced use case that Tony ran into.

Farshid Ghods added a comment - 17/Aug/12 3:48 PM http://10.1.3.67:8091 saslbucket saslpass

Aleksey Kondratenko added a comment - 17/Aug/12 5:11 PM no it's not duplicate. That's merely timeout

Aleksey Kondratenko added a comment - 17/Aug/12 5:12 PM I'll take a quick look but quick. If it it's remotely similar to paging issues we're already aware of then I'll switch to functional bugs which we have plenty for now

Aleksey Kondratenko added a comment - 17/Aug/12 5:25 PM I can fix those timeouts by: *) if consistent views are not enabled don't touch index pausing APIs *) set update_vbucket_state timeout to infinity anticipating lots of indexes Please allow for 20c

Aleksey Kondratenko added a comment - 17/Aug/12 5:27 PM See above

Farshid Ghods added a comment - 17/Aug/12 5:46 PM before we continue with any fix i would like to understand why this happens. is this because of memcached process slowness responding to stats call ? also about the options you provided 1) we are not using consistent views here on this build so why bother changing any configuration there ? 2) did this logic change recently ? what do you mean by lot of indexes. in a typical case for customers with 10+ indexes there will be indexing during rebalancing and yes we have to handle waiting or shutting down those indexes 3) does this vbucket_state include views or only a call to memcached plus this is such a basic use case : 400k items , 2 nodes , memcached size 500m and beam.smp is 500 mb

Aleksey Kondratenko added a comment - 17/Aug/12 6:17 PM Most likely set views code is not bothering itself with prompt reply on unpause indexes call. I'm seeing 4-5 seconds times from calling this. And this was added as part of consistent views work. Which is disabled, but side effect is those useless unpause_indexes calls that basically pass all vbuckets all the time.

Aleksey Kondratenko added a comment - 17/Aug/12 6:17 PM update_vbucket_state includes both views changes and memcached vbucket state and possibly replication source change

Farshid Ghods added a comment - 17/Aug/12 6:27 PM - edited okay so may i propose reverting those changes ? and go back to this commit where consistent view work was not merged ?

Aleksey Kondratenko added a comment - 17/Aug/12 6:45 PM I'd like to avoid that. I'll remove that effect without reverting everything

Peter Wansch added a comment - 17/Aug/12 7:04 PM Alk, as discussed, please go ahead and make these changes for 2.0.0c: *) if consistent views are not enabled don't touch index pausing APIs *) set update_vbucket_state timeout to infinity anticipating lots of indexes

Thuan Nguyen added a comment - 18/Aug/12 2:11 AM Integrated in github-ns-server-2-0 #440 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/440/ ])      MB-6284 : bumped set_vbucket_state_timeout to infinity (Revision 860d46fa549010ec09e80f6e292f50b2cb72bccd) MB-6284 : don't touch index pausing API api if we don't need it (Revision 1d0365346ef9d22d4ffbc4e054ad63235165d418)      Result = SUCCESS pwansch : Files : * src/janitor_agent.erl pwansch : Files : * src/capi_set_view_manager.erl * src/cluster_compat_mode.erl

Farshid Ghods added a comment - 18/Aug/12 5:29 PM rebalance failures are not reported anymore after the fix was merged so marking this as fixed. Thanks.

Thuan Nguyen added a comment - 20/Aug/12 5:12 PM Integrated in github-ns-server-2-0 #444 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/444/ ])      MB-6284 : actually check for index_pausing_disabled variable (Revision ac0409b663ad844a58f6119018e6a93c9cbba149)      Result = SUCCESS Aliaksey Kandratsenka : Files : * src/cluster_compat_mode.erl

Thuan Nguyen added a comment - 24/Aug/12 12:14 PM At timeout set to infinity, I still hit this bug again in build 2.0.0-1628 when I do failover node 25 and remove node 26 at the same time. I will attach atop, collect info soon. Cluster setup with 11 nodes, 2 3GB bucket, 8 docs (4 per bucket), 2 views per doc 10.3.121.13 10.3.121.14 10.3.121.15 10.3.121.16 10.3.121.17 10.3.121.20 10.3.121.22 10.3.121.26 10.3.121.24 10.3.121.25 10.3.121.23 Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{' ns_1@10.3.121.20 ', {'EXIT', {noproc, {gen_server,call, [{'janitor_agent-default-s', ' ns_1@10.3.121.20 '}, {if_rebalance,<0.17596.50>, {update_vbucket_state,389,replica, undefined,' ns_1@10.3.121.22 '}}, infinity]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}

Thuan Nguyen added a comment - 24/Aug/12 2:42 PM Link to atop file https://s3.amazonaws.com/packages.couchbase/atop-files/2.0.0/atop-node13 https://s3.amazonaws.com/packages.couchbase/atop-files/2.0.0/atop-node20 https://s3.amazonaws.com/packages.couchbase/atop-files/2.0.0/atop-node22 Diags from all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/11nodes-1628-reb-bulk_set_vbucket_state_failed-2120824.tgz collect_info of 10 nodes https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_0/10nodes-col-inf-1628-reb-bulk_set_vbucket_state_failed-2120824.tgz collect_info at node 17 failed due to bug MB-6414

Aleksey Kondratenko added a comment - 27/Aug/12 1:10 PM Please file separate ticket. This is a noproc crash. And in fact looks similar to MB-6415

Abhinav Dangeti added a comment - 28/Aug/12 5:40 PM Issue still persists on build 1649. Source: 10.1.3.235, 10.1.3.236, 10.3.2.54 Destination: 10.1.3.237, 10.1.3.238, 10.3.2.55 In a 3:3 unidirectional set up, with on going replication, when I tried to rebalance out one of the nodes (10.3.2.55) on the destination cluster, I saw that replication failed: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{' ns_1@10.1.3.238 ', {'EXIT', {{timeout, {gen_server,call, ['ns_memcached-bucket', {set_vbucket,216,active}, 60000]}}, {gen_server,call, [{'janitor_agent-bucket', ' ns_1@10.1.3.238 '}, {if_rebalance,<0.11867.32>, {update_vbucket_state,731,replica, undefined,' ns_1@10.1.3.237 '}}, infinity]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I got the grabdiags of the 6 nodes, and they are available at the following links: https://s3.amazonaws.com/bugdb/MB-6462/10.1.3.235-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/MB-6462/10.1.3.236-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/MB-6462/10.1.3.237-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/MB-6462/10.1.3.238-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/MB-6462/10.3.2.54-8091-diag.txt.gz https://s3.amazonaws.com/bugdb/MB-6462/10.3.2.55-8091-diag.txt.gz

Ketaki Gangal added a comment - 28/Aug/12 7:14 PM Assigning all time-out bugs to be re-assigned further. Seeing numerous timeouts during rebalancing, need a dedicated owner to triage/fix this.

Farshid Ghods added a comment - 29/Aug/12 10:12 AM Ketaki Per alk comments please open a new ticket for this rebalance failure instead of reopening a similar ticket

Ketaki Gangal added a comment - 29/Aug/12 12:19 PM Hi Aleksey, Is this issue with "noproc" on this bug? or bug http://www.couchbase.com/issues/browse/MB-6428

Thuan Nguyen added a comment - 29/Aug/12 2:12 PM Alk, I got rebalance timeout error, 2012-08-29 01:06:21.275 ns_orchestrator:2:info:message( ns_1@10.3.121.13 ) - Rebalance exited with reason {{bulk_set_vbucket_state_failed,                                [{' ns_1@10.3.121.20 ',                                  {'EXIT',                                   {{timeout,                                     {gen_server,call,                                      [timeout_diag_logger,                                       {diag,                                        {timeout,                                         {gen_server,call,[ns_config,get]}}}]}},                                    {gen_server,call,                                     [{'janitor_agent-saslbucket',                                       ' ns_1@10.3.121.20 '},                                      {if_rebalance,<0.12846.432>,                                       {update_vbucket_state,294,replica,                                        undefined,undefined}},                                      infinity]}}}}]},                               [{janitor_agent,bulk_set_vbucket_state,4},                                {ns_vbucket_mover,                                 update_replication_post_move,3},                                {ns_vbucket_mover,handle_info,2},                                {gen_server,handle_msg,5},                                {proc_lib,init_p_do_apply,3}]} Is it the same timeout error with this bug or different one?

Farshid Ghods added a comment - 30/Aug/12 1:28 PM Abhinav/Ketaki, we are not able to make any progress on timeout related bugs unless we have atop command running during this process to confirm there is no swapping or paging problem. also please post more information about number of items , ram usage , cpu usage , memory usage or simply post atop output to the bug

Ketaki Gangal added a comment - 30/Aug/12 6:26 PM Yes, will add more logs on this.

Farshid Ghods added a comment - 03/Sep/12 5:32 PM is this issue related to xdcr ?

Junyi Xie added a comment - 03/Sep/12 5:59 PM Hi Fashid, xdcr is obviously a victim of this bug. XDCR itself will not be able to break rebalance process at destination cluster.

Farshid Ghods added a comment - 05/Sep/12 2:10 PM Andrei, are you able to reproduce this ?

Andrei Baranouski added a comment - 05/Sep/12 2:26 PM I can't reproduce it, tried manually with the similar steps above( didn't try xdcr scenarious)

Junyi Xie added a comment - 05/Sep/12 2:54 PM Based on Andrei's test

Ketaki Gangal added a comment - 05/Sep/12 3:11 PM Andrei has tested this on a non-xdcr setup. @Andrei: Can you test this on a 3:3 unidireciotnal replication with 1M items? rebalance out nodes at destination?

Andrei Baranouski added a comment - 05/Sep/12 9:11 PM checked different combination, it was not reproduced with bulk_set_vbucket_state_failed error. I will close the bug, new one was open MB-6535 Rebalance stopped by janitor on destination cluster and then master has changed several times

Andrei Baranouski added a comment - 05/Sep/12 9:11 PM is not reproduced anymore