addback after autofailover due to reboot causes rebalance failure

Aleksey Kondratenko added a comment - 25/Apr/12 1:09 PM There's something very weird with environment here. Node .99 thinks it's .100 when .100 is actually clearly different node.

Farshid Ghods added a comment - 25/Apr/12 7:05 PM Keith, please reproduce this on a different environment

Keith Batten added a comment - 26/Apr/12 4:30 PM reproduced. including an erlang crash dump

Aleksey Kondratenko added a comment - 26/Apr/12 11:03 PM Diags of .104 (ehostunreach node) are missing. Will see if I can find something without logs.

Aleksey Kondratenko added a comment - 26/Apr/12 11:06 PM erl_crash.dump is from Apr 20.

Aleksey Kondratenko added a comment - 26/Apr/12 11:08 PM I need more details. Please, logs from all nodes. As well as fresh description of what exactly was done in new case if it's different even in tiny details from original description. Also, please, double check your environment. Particularly, when node is added to cluster it will check connectivity to node it was added to. But not all other nodes.

Aleksey Kondratenko added a comment - 26/Apr/12 11:24 PM Hm. Sorry I was wrong about crash dump. It's today's. Still I really need logs from that failed node in order to solve this case. And looking at this further that's probably not environment.

Aleksey Kondratenko added a comment - 26/Apr/12 11:26 PM And also please check that this node that still thinks it's .104 is really accessible from outside as .104. Maybe it's environment problem.

Keith Batten added a comment - 27/Apr/12 3:02 PM I was able to get the browse_logs output from 104, diags are unavailable due to the node being down.

Aleksey Kondratenko added a comment - 27/Apr/12 4:15 PM Please, explain exactly what happened to that bad node. Erlang crashed? Entire machine went down? Any traces of erlang crash dump ?

Aleksey Kondratenko added a comment - 27/Apr/12 4:21 PM Apparently erlang did not crash. I need very detailed description of what happened before/after node was restarted. Details of what was done and environment seems to be really important to understand whats going on.

Keith Batten added a comment - 27/Apr/12 6:41 PM On the bad node erlang is unable to start back up after the reboot (attempted service couchbase-server restart, but it didn't work.) Let me clarify, the node that fails is the same one that was rebooted. 1) cluster 4 nodes using the web console on 125 2) create default bucket 3) enable and set autofailover to 30s 4) reboot node 104 5) watch web console to see when 104 is autofailed, and then when it shows the "Add back" option. 6) click "Add back" 7) click "Rebalance"

Aleksey Kondratenko added a comment - 27/Apr/12 6:43 PM So reboot takes longer than 30 seconds?

Aleksey Kondratenko added a comment - 27/Apr/12 6:48 PM May I have access ?

Keith Batten added a comment - 27/Apr/12 7:05 PM The reboot takes maybe around a minute for everything to come back up. Yeah you can have access to those nodes.

Aleksey Kondratenko added a comment - 30/Apr/12 11:31 AM May I ask for yet another reproduction of this and credentials to access this bad node ?

Aleksey Kondratenko added a comment - 30/Apr/12 2:31 PM Ok. Here's what I've found on that host: [ root@localhost ~]# iptables-save # Generated by iptables-save v1.3.5 on Mon Apr 30 07:24:49 2012 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [138758:81365428] :RH-Firewall-1-INPUT - [0:0] -A INPUT -j RH-Firewall-1-INPUT -A FORWARD -j RH-Firewall-1-INPUT -A RH-Firewall-1-INPUT -i lo -j ACCEPT -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT -A RH-Firewall-1-INPUT -p esp -j ACCEPT -A RH-Firewall-1-INPUT -p ah -j ACCEPT -A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Mon Apr 30 07:24:49 2012 [ root@localhost ~]#

Aleksey Kondratenko added a comment - 30/Apr/12 2:32 PM which means only connections to port 631 (cups) and 22 (sshd) are allowed.

Keith Batten added a comment - 30/Apr/12 3:00 PM I'm closing this as invalid then. I think at most there needs to be a second check to verify that the cluster can communicate with the failed over node before it shows "add back".

Aleksey Kondratenko added a comment - 30/Apr/12 3:05 PM The issue is that _that_ node can contact rest of cluster. Reverse is not true. And indeed it can cause some issues for somebody.

Farshid Ghods added a comment - 30/Apr/12 3:12 PM we should not close this ticket. ns-server should not show the node as being ready to be added back

Aleksey Kondratenko added a comment - 30/Apr/12 3:30 PM Node _was_ able to connect to rest of cluster. We cannot handle any possible misconfiguration. Please open feature request instead for this particular case

Farshid Ghods added a comment - 30/Apr/12 3:34 PM Aleksey Kondratenko added a comment - 30/Apr/12 3:30 PM Node _was_ able to connect to rest of cluster. We cannot handle any possible misconfiguration. Please open feature request instead for this particular case okay but you also mentioned that Aleksey Kondratenko added a comment - 30/Apr/12 3:05 PM >>The issue is that _that_ node can contact rest of cluster. Reverse is not true. maybe im missing something here so why do we allow the user to add the node back into the cluster if some nodes in the cluster are not able to connect to this node ?

Aleksey Kondratenko added a comment - 30/Apr/12 3:52 PM - edited Erlang doesn't establish connection twice. If some node contacted some other node, that's bidirectional connection that will be used between this pair by erlang. So checking if both node can connect to each other requires extra code. And then we can still invent myriad of ways how things can be misconfigured with us not detecting it. And then my point is: we cannot check for every possible misconfiguration. Also this is not a bug but a "lack" of feature. So I propose filing new "feature request" and making sure it gets Dipti's attention.

Peter Wansch added a comment - 19/Jun/12 7:30 PM Assigning back to keep it in the triage process for now.

Aleksey Kondratenko added a comment - 16/Oct/12 8:51 AM While there's obviously good thing about us diagnosing this case better, there's _thousands_ of ways any particular environment may have environment issues that would prevent us from working properly. I think it's better to keep this off our table at least for now. We have better things to fix IMHO.