Home | Forums | Membase | Memcached Server 1.0.3

How to add a previously removed memcached node to an existing cluster?

1 reply [Last post]
Mon, 07/19/2010 - 07:12
rally25rs
Offline
Joined: 06/11/2010
Groups: None

While trying to diagnose another issue, I think I may have created another. This is on a production NS cluster, so I want to make sure I don't mess up the entire cache cluster...

I had an existing 2-node cluster. Lets call them Server-A and Server-B.

The web console of both servers showed both Server-A and Server-B's memcached nodes.

In the web console's Server Nodes section, I clicked 'remove from cluster' on Server-B's memcached node. This (obviously) removed it from the cluster.

Now, the web console on Server-A and Server-B both show only Server-A's memcached node, and show analytics for the single node cluster (in other words, the 2 web consoles are still clustered, with only Server-A's memcached node).

However, both web consoles show the "Join Cluster" button, as if they were not clustered.

On Server-B, the memcached node that was removed from the cluster is still running (still listening on port 11211). Neither of the web consoles shows Server-B's memcached node in their server list though (which is correct, since its the node that I removed).

So what I want to do is add Server-B's memcached node back into the cluster. I think I need to click 'Join Cluster' in Server-B's web console, but Server-B's web console shows Server-A's memcached node in its server nodes list, not its own local memcached. I'm afraid that it will try to add Server-A's memcached to the cluster that it is already a part of.

So... what the heck did I do wrong here and how to I fix it, without interrupting an app running in production?

Thanks for any help!

Top
Mon, 07/19/2010 - 08:33
ingenthr
Offline
Joined: 03/16/2010
Groups:

I believe you are not using a "smart client", correct? So you still have workload going to both servers? That affects things, as the smart client will "see" the cluster the same way the web console does.

Doing a "join" from B to A as you suggest should give you the right behavior. The showing of the incorrect server in the server list is a bug. It is an intermittent bug though. If that does not give you the right behavior, it may be necessary to stop services, remove the existing config, then start services on node B. This will generate a new "blank" configuration which should be able to join the cluster. The best way to do this is to move or rename the "config.dat" under your installation.

Note that the [I]joinee[/I] will pick up the bucket configuration of the cluster it is joining, so you should be able to retain all of your existing configuration if it's on node A.

It goes mostly without saying, but because this is a production cluster, I'd recommend finding a slow point or maintenance interval to carry out these activities.

Top