Can not rebalance
Hi
I'm adding a membasenode to a cluster by means of the REST API and at the same time updating firewall settings for all nodes. All works fine and I can see the node is added to the cluster. However the final step which I do manually (rebalance, which I suppose is also possible with REST) fails to do so and I suspect the firewall is to blame. This is what I have:
Chain INPUT (policy ACCEPT)
ACCEPT tcp -- anywhere anywhere tcp dpt:8091
ACCEPT tcp -- 192.168.178.43 anywhere multiport dports 4369,11210,11211
ACCEPT tcp -- 192.168.178.43 anywhere tcp dpts:21100:21199
ACCEPT udp -- 192.168.178.43 anywhere udp dpt:11210
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
... where 192.168.178.43 is the other node. Of course 192.168.178.43 also has a firewall similar to the above with the other node IP.
I can telnet 11210 from both nodes which I believe is used by membase rebalance etc. right? 11210 seems to be the problem according to the log:
Rebalance exited with reason {mover_failed,{abnormal,78}}
ns_orchestrator002 00:06:17 - Tue Jun 7, 2011
Port server vbucketmigrator on node 'ns_1@192.168.148.229' exited with status 78. Restarting. Messages: Authenticated towards: {Sock 192.168.178.43:11210}
Connecting to {Sock 192.168.148.229:11210}
Failed to connect to host: Failed to connect to [192.168.148.229:11210] ns_port_server000 00:06:17 - Tue Jun 7, 2011
Port server vbucketmigrator on node 'ns_1@192.168.148.229' exited with status 78. Restarting. Messages: Authenticated towards: {Sock 192.168.178.43:11210}
Connecting to {Sock 192.168.148.229:11210}
Failed to connect to host: Failed to connect to [192.168.148.229:11210] (repeated 1 times) ns_port_server000 00:05:50 - Tue Jun 7, 2011
Port server vbucketmigrator on node 'ns_1@192.168.148.229' exited with status 78. Restarting. Messages: Authenticated towards: {Sock 192.168.178.43:11210}
Connecting to {Sock 192.168.148.229:11210}
Failed to connect to host: Failed to connect to [192.168.148.229:11210] ns_port_server000 23:59:58 - Mon Jun 6, 2011
Starting rebalance, KeepNodes = ['ns_1@192.168.148.229','ns_1@192.168.178.43'], EjectNodes = []
Thanks
Hi
All fine now I needed an additional NEW so this:
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
... became this:
ACCEPT all -- anywhere anywhere state NEW,RELATED,ESTABLISHED
I'm still in the wild as to how to, with REST, achieve adding/joining a node to the cluster and rebalance it afterwards. I see no documentation describing this?
Doing a curl http://localhost:8091/controller/rebalance returns a "Not found." from the bash command line where the node resides.
Thanks...
All our information on the REST API is located here: http://www.couchbase.org/wiki/display/membase/Membase+Management+REST+API
Maybe you've found that already, I know there isn't a whole lot of examples. It would actually be really great if you could take all the trouble that you've gone through and update our documentation where you feel it is lacking.
I would also very much support you creating a "Membase and IPtables" page since you're clearly well versed not only in troubleshooting but also managing iptables.
Lastly, you might want to use our CLI interface rather than the REST API directly since a) it has better help and examples and b) any future changes we make to the REST API will be masked by the CLI interface.
Let me know if that helps, thanks again for your patience.
Perry
The given here info is very useful, membaseuser_dk! Thanks for sharing! Stuart Alexander, manager of family tree maker free
Hi again
If looking at the POST data (http://extip:8091/controller/rebalance) I see the parameters are
ejectedNodes
knownNodes ns_1@192.168.148.229,ns_1@192.168.178.43
... and response is empty.
It is not all clear to me how to achieve adding/joining a node to cluster and perform rebalancing with REST (perhaps in one operation), maybe anyone can shed light on that?
Thanks..