Rebalance fail and more

That looks like MB-7108, but this was fixed in 2.0 GA.  Is this with the current GA release?  Well, match also seems to match a few other rebalance failures.

To address your questions...

What should I do to resolve this and upgrade all my nodes?

I'd recommend looking at how the vbuckets are distributed through the cluster through the Web-UI.  Hopefully, whenever you request a rebalance we're making progress.  Rebalance may fail, but if we're successfully moving some vbuckets before that failure, we can get there eventually.  This isn't expected, but we can work our way to resolution from here.

also why is the cluster no replicating the items?

I don't have a good explanation for this.  Can you possibly do a cbcollect_info and upload it per the directions here.  We'll get someone to have a look at it.

Rebalancing doesn't seem to do anything, item count per node stays the same as weel as active vBuckets count in the resources section.
also rebalance fails even without adding a 2.0 node - with a cluster of all 1.8.1 nodes.
I've uploaded the cbstats from all nodes (ignore the .info file I uploaded - it's a duplicate of one of the nodes).

Thanks for the help!

Sorry to hear it's staying the same. The first change we'll want to try is tuning some timeout values a bit higher to give this processing a chance to complete rebalance.

This is a bit low level, but I can help you through it.  What's been recommended to me is extending a number of different timeout values.  You can do this with this kind of approach:

wget -O- --user=Administrator --password=password --post-data='ns_config:set({node, node(), {timeout, ns_memcached_outer_very_heavy}}, 120000).' http://10.3.2.83:8091/diag/eval|http://10.3.2.83:8091/diag/eval

This would adjust 'ns_memcached_outer_very_heavy'.  Now, where to adjust these to depends quite a bit on the environment.  For now, let's double them in your environment to see if that helps us get further with rebalance.  Using the wget style request outlined above, adjust them with these values.

1. ns_memcached_outer, 120000
2. ns_memcached_outer_heavy, 120000
3. ns_memcached_outer_very_heavy, 240000
4. ns_memcached_open_checkpoint, 120000
5. ns_memcached_connected, 10000
6. ebucketmigrator_connect, 120000

 This can be done on the live running cluster. It takes effect immediately.  Then retry the rebalance.

Let us know how this goes.

Thanks, but I cant get that command to work properly
when I type it as you entered it , with changing the ip and password, I get this response

 
-bash: <a href="http://10.28.91.176:8091/diag/eval:" title="http://10.28.91.176:8091/diag/eval:">http://10.28.91.176:8091/diag/eval:</a> No such file or directory
--2013-01-03 10:12:08--  <a href="http://10.28.91.176:8091/diag/eval<br />
Connecting" title="http://10.28.91.176:8091/diag/eval<br />
Connecting">http://10.28.91.176:8091/diag/eval<br />
Connecting</a> to 10.28.91.176:8091... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 10.28.91.176:8091... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2
Saving to: `STDOUT'
 
 0% [                                                                                                                                                                 ] 0           --.-K/s   in 0s      
 
 
Cannot write to `-' (Broken pipe).

and when I omit the -O- tag from the command, I get this :

-bash: <a href="http://10.29.237.181:8091/diag/eval:" title="http://10.29.237.181:8091/diag/eval:">http://10.29.237.181:8091/diag/eval:</a> No such file or directory
--2013-01-03 10:18:41--  <a href="http://10.29.237.181:8091/diag/eval<br />
Connecting" title="http://10.29.237.181:8091/diag/eval<br />
Connecting">http://10.29.237.181:8091/diag/eval<br />
Connecting</a> to 10.29.237.181:8091... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 10.29.237.181:8091... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2
Saving to: `eval.5'

since I see the Unauthorized error, I am guessing it fails, what am I doing wrong?
should I run this script on every node in the cluster or are these cluster wide settings?
Thanks and a Happy new year!
Yuval

The 401 response would be owing to invalid credentials. Either the Administrator username is different or the password is different in your case. Do keep in mind, special characters may need to be quoted, if you have them in the username or password.

Once you get this working, it needs only be done on one node of the cluster. The change should propagate to all other nodes.

if I type my credentials wrong I get an "Authentication failed" message as followed, so I guess I am entering them correctly

-bash: <a href="http://10.29.237.181:8091/diag/eval:" title="http://10.29.237.181:8091/diag/eval:">http://10.29.237.181:8091/diag/eval:</a> No such file or directory
--2013-01-04 11:22:40--  <a href="http://10.29.237.181:8091/diag/eval<br />
Connecting" title="http://10.29.237.181:8091/diag/eval<br />
Connecting">http://10.29.237.181:8091/diag/eval<br />
Connecting</a> to 10.29.237.181:8091... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Connecting to 10.29.237.181:8091... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authorization failed.

I'm not certain why. I'll have a colleague look into it and post back.

Hi it appears to be that the above error return (401) was due to an authentication failure. Maybe typos in user id/password? Please verify if all was correct.
The rest error returns (first two 401 responses from earlier posts) seem to be normal behavior. Given the wget command it first attempted to send the request without password then immediately succeeded during the second attempt with password. Thus the particular timeout value must have gotten changed correctly. Thanks for your time and help!

Thanks Jin,
Well if thats the case and I changed the values correctly the first time, it did not solve the Rebalance issues I've been having. what should I do next?

Hi,
I've decided to drop the cluster and install a fresh new one before going into production. so I installed 2 Couchbase community servers on AWS EC2s, using the rpm package and not the AMI. everything worked fine but no replication was going on and adding a new server caused rebalance to fail again, this time with a "no reply" message from the other servers. the third node was added successfully but it did not receive any new items and not finishing rebalance just like my 1.8 installation.
so I did a fresh install using the Community server AMI this time, but the data library that is set there by default is not pointing to the EBS the AMI installed.
so I had to change the ownership of that volume and then setting it up in the setup dialog. which is not noted anywhere in your docs.
Now adding new nodes and rebalancing works.
So my guess is that there was something wrong with the way I've configured the addresses in my previous faulty installation
I did it by modifying /opt/couchbase/bin/couchbase-server and adding the private ip from amazon. I've noticed that the AMI installation does not add the ip address to the couchbase-server file
how does the AMI installation configures the ip addresses?

Hi thanks much for your detailed step-by-step explanation. We will bring your info to right support and get back to you soon.