Couchbase

Couchbase http://www.couchbase.com/issues This file is an XML representation of an issue en-us

5.2.4

845

26-12-2012

[MB-5602] auto-failover fails over a node if some of the buckets are already rebalanced out but rebalance has been stopped or interrupted ( auto-failover should failover if all buckets are down) http://www.couchbase.com/issues/browse/MB-5602 Couchbase Server Setup <br/> 1.Setup a 18 node cluster with 2 buckets- bucket1, bucket2 <br/> 2. Enable auto-failover <br/> 3. Add a new node 126 <br/> 4. Rebalance <br/> <br/> Output <br/> 1. Rebalance works fine. But seeing these log messages - <br/> <br/> Could not automatically failover node '<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>>>' because I think rebalance is running auto_failover000 <a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>>> 19:32:12 - Sun Jun 17, 2012 <br/> Bucket "bucket1" loaded on node '<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>>>' in 0 seconds. ns_memcached001 <a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>>> 19:32:04 - Sun Jun 17, 2012 <br/> Started rebalancing bucket bucket2 ns_rebalancer000 <a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>>> 19:31:36 - Sun Jun 17, 2012 <br/> Starting rebalance, KeepNodes = ['<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a><mailto:<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a>><mailto:<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a><mailto:<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a>>>','<a href='mailto:ns_1@10.3.2.86'>ns_1@10.3.2.86</a><mailto:<a href='mailto:ns_1@10.3.2.86'>ns_1@10.3.2.86</a>><mailto:<a href='mailto:ns_1@10.3.2.86'>ns_1@10.3.2.86</a><mailto:<a href='mailto:ns_1@10.3.2.86'>ns_1@10.3.2.86</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.87'>ns_1@10.3.2.87</a><mailto:<a href='mailto:ns_1@10.3.2.87'>ns_1@10.3.2.87</a>><mailto:<a href='mailto:ns_1@10.3.2.87'>ns_1@10.3.2.87</a><mailto:<a href='mailto:ns_1@10.3.2.87'>ns_1@10.3.2.87</a>>>','<a href='mailto:ns_1@10.3.2.88'>ns_1@10.3.2.88</a><mailto:<a href='mailto:ns_1@10.3.2.88'>ns_1@10.3.2.88</a>><mailto:<a href='mailto:ns_1@10.3.2.88'>ns_1@10.3.2.88</a><mailto:<a href='mailto:ns_1@10.3.2.88'>ns_1@10.3.2.88</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.89'>ns_1@10.3.2.89</a><mailto:<a href='mailto:ns_1@10.3.2.89'>ns_1@10.3.2.89</a>><mailto:<a href='mailto:ns_1@10.3.2.89'>ns_1@10.3.2.89</a><mailto:<a href='mailto:ns_1@10.3.2.89'>ns_1@10.3.2.89</a>>>','<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a><mailto:<a href='mailto:ns_1@10.3.2.104'>ns_1@10.3.2.104</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.105'>ns_1@10.3.2.105</a><mailto:<a href='mailto:ns_1@10.3.2.105'>ns_1@10.3.2.105</a>><mailto:<a href='mailto:ns_1@10.3.2.105'>ns_1@10.3.2.105</a><mailto:<a href='mailto:ns_1@10.3.2.105'>ns_1@10.3.2.105</a>>>','<a href='mailto:ns_1@10.3.2.106'>ns_1@10.3.2.106</a><mailto:<a href='mailto:ns_1@10.3.2.106'>ns_1@10.3.2.106</a>><mailto:<a href='mailto:ns_1@10.3.2.106'>ns_1@10.3.2.106</a><mailto:<a href='mailto:ns_1@10.3.2.106'>ns_1@10.3.2.106</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a><mailto:<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>><mailto:<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a><mailto:<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>>>','<a href='mailto:ns_1@10.3.2.109'>ns_1@10.3.2.109</a><mailto:<a href='mailto:ns_1@10.3.2.109'>ns_1@10.3.2.109</a>><mailto:<a href='mailto:ns_1@10.3.2.109'>ns_1@10.3.2.109</a><mailto:<a href='mailto:ns_1@10.3.2.109'>ns_1@10.3.2.109</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.110'>ns_1@10.3.2.110</a><mailto:<a href='mailto:ns_1@10.3.2.110'>ns_1@10.3.2.110</a>><mailto:<a href='mailto:ns_1@10.3.2.110'>ns_1@10.3.2.110</a><mailto:<a href='mailto:ns_1@10.3.2.110'>ns_1@10.3.2.110</a>>>','<a href='mailto:ns_1@10.3.2.111'>ns_1@10.3.2.111</a><mailto:<a href='mailto:ns_1@10.3.2.111'>ns_1@10.3.2.111</a>><mailto:<a href='mailto:ns_1@10.3.2.111'>ns_1@10.3.2.111</a><mailto:<a href='mailto:ns_1@10.3.2.111'>ns_1@10.3.2.111</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.112'>ns_1@10.3.2.112</a><mailto:<a href='mailto:ns_1@10.3.2.112'>ns_1@10.3.2.112</a>><mailto:<a href='mailto:ns_1@10.3.2.112'>ns_1@10.3.2.112</a><mailto:<a href='mailto:ns_1@10.3.2.112'>ns_1@10.3.2.112</a>>>','<a href='mailto:ns_1@10.3.2.113'>ns_1@10.3.2.113</a><mailto:<a href='mailto:ns_1@10.3.2.113'>ns_1@10.3.2.113</a>><mailto:<a href='mailto:ns_1@10.3.2.113'>ns_1@10.3.2.113</a><mailto:<a href='mailto:ns_1@10.3.2.113'>ns_1@10.3.2.113</a>>>', <br/> '<a href='mailto:ns_1@10.3.2.114'>ns_1@10.3.2.114</a><mailto:<a href='mailto:ns_1@10.3.2.114'>ns_1@10.3.2.114</a>><mailto:<a href='mailto:ns_1@10.3.2.114'>ns_1@10.3.2.114</a><mailto:<a href='mailto:ns_1@10.3.2.114'>ns_1@10.3.2.114</a>>>','<a href='mailto:ns_1@10.3.2.115'>ns_1@10.3.2.115</a><mailto:<a href='mailto:ns_1@10.3.2.115'>ns_1@10.3.2.115</a>><mailto:<a href='mailto:ns_1@10.3.2.115'>ns_1@10.3.2.115</a><mailto:<a href='mailto:ns_1@10.3.2.115'>ns_1@10.3.2.115</a>>>', <br/> '<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>>>'], EjectNodes = [] <br/> <br/> Attached are the web-logs and logs from master node-104. <br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/ns-diag-20120618095246.txt">https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/ns-diag-20120618095246.txt</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/10.3.2.104-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/10.3.2.104-8091-diag.txt.gz</a> <br/> <br/> <br/> Other related conversation <br/> I have enabled auto-failover on the large-cluster and every time I rebalance In a node, I get an error message showing " Could not automatically failover node '<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a><mailto:<a href='mailto:ns_1@10.3.121.126'>ns_1@10.3.121.126</a>>>' because I think rebalance is running" . <br/> The node 126 is newly added and rebalance issued, is this message displayed because the node is not yet ready to join the cluster ? <br/> The rebalance works fine, but I do not understand why is auto-failover attempted in here. Any idea? <br/> <br/> No. according to logs at 19:32:04 bucket1 was loaded. Maybe there are some other buckets that are still not ready on this node. May I have logs? <br/> <br/> 18 node cluster, Centos <br/> Build 181-918 <br/> 2 buckets, 1024vbuckets MB-5602

auto-failover fails over a node if some of the buckets are already rebalanced out but rebalance has been stopped or interrupted ( auto-failover should failover if all buckets are down)

Bug Blocker Closed Fixed Public Aleksey Kondratenko Ketaki Gangal Mon, 18 Jun 2012 16:55:39 -0500 Wed, 9 Jan 2013 22:59:34 -0600 Tue, 19 Jun 2012 18:53:16 -0500 1.8.1-release-candidate 1.8.1 ns_server 0 1 <br/> This happens if you have 2 or more buckets and autofailover is enabled and if both buckets have significant amount of data. <br/> <br/> After rebalancing out first bucket node will incorrectly be interpreted as down by autofailover service since it doesn't have all buckets this service thinks (incorrectly) it needs to have. <br/> <br/> Normally rebalance prevents autofailover to actually do anything, but if rebalance is stopped, then 'partially' rebalance out node will be automatically failed over. <br/> <br/> Seen here: <a href="https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/10.3.2.104-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/web-log-largeCluster/10.3.2.104-8091-diag.txt.gz</a> <br/> <a href="http://review.couchbase.org/#change,17372">http://review.couchbase.org/#change,17372</a> Tested on large-cluster build 927- Not seeing anymore auto-failover messages. <br/> Closing this bug for now. Integrated in github-ns-server-2-0 #380 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/380/">http://qa.hq.northscale.net/job/github-ns-server-2-0/380/</a>])<br/>     <a href="http://www.couchbase.com/issues/browse/MB-5602" title="auto-failover fails over a node if some of the buckets are already rebalanced out but rebalance has been stopped or interrupted ( auto-failover should failover if all buckets are down)"><strike>MB-5602</strike></a>: consider buckets' servers list when computing down nodes (Revision 72b674c47e386dac5a28ecaadfea2f37c3d14133)<br/> <br/>      Result = SUCCESS<br/> Farshid Ghods : <br/> Files : <br/> * src/auto_failover.erl<br/> Date of First Response Mon, 18 Jun 2012 17:14:49 -0500 Rank 4928 Time In Status