How to add a previously removed memcached node to an existing cluster?
While trying to diagnose another issue, I think I may have created another. This is on a production NS cluster, so I want to make sure I don't mess up the entire cache cluster...
I had an existing 2-node cluster. Lets call them Server-A and Server-B.
The web console of both servers showed both Server-A and Server-B's memcached nodes.
In the web console's Server Nodes section, I clicked 'remove from cluster' on Server-B's memcached node. This (obviously) removed it from the cluster.
Now, the web console on Server-A and Server-B both show only Server-A's memcached node, and show analytics for the single node cluster (in other words, the 2 web consoles are still clustered, with only Server-A's memcached node).
However, both web consoles show the "Join Cluster" button, as if they were not clustered.
On Server-B, the memcached node that was removed from the cluster is still running (still listening on port 11211). Neither of the web consoles shows Server-B's memcached node in their server list though (which is correct, since its the node that I removed).
So what I want to do is add Server-B's memcached node back into the cluster. I think I need to click 'Join Cluster' in Server-B's web console, but Server-B's web console shows Server-A's memcached node in its server nodes list, not its own local memcached. I'm afraid that it will try to add Server-A's memcached to the cluster that it is already a part of.
So... what the heck did I do wrong here and how to I fix it, without interrupting an app running in production?
Thanks for any help!