Couchbase cluster reconfiguration
I've got a Couchbase(2.0.1) cluster with 6 nodes(3 replicas). On the client side I use a Moxi server(1.8.1) and memcached clients. I've encountered an issue with this setup. When a cluster node goes down, the Moxi server continues considering this node up and tries to write to it. As I read in the Moxi server documentation, a Moxi server communicates with a cluster through HTTP/REST and gets an actual cluster configuration and, when the configuration changes, the cluster notifies connected Moxi servers about reconfiguration. So in my case, it seems that the Moxi server is not getting reconfiguration messages.
For the sake of curiosity, I tried to connect to the cluster directly through libcouchbase(2.0.7) and went to the same problem. When a cluster node goes down, the library still tries to write to this node as the Moxi server did.
The Moxi server configuration:
port_listen=8095, downstream_max=1024, downstream_conn_max=1, downstream_conn_queue_timeout=200, downstream_timeout=5000, wait_queue_timeout=200, connect_max_errors=5, connect_retry_interval=30000, connect_timeout=400, auth_timeout=100, cycle=200
First of all your question is not related to Moxi or Libcouchbase, this is a generic question regarding Couchbase cluster behavior.
With Couchbase when a node is down you have to do a fail over, to promote the replicas to active, and then all the clients will be alerted and will use the keys/documents from the existing nodes. This means that until you do a fail over (manually or automatically) the data that are on this nodes are not accessible and you can not save data into it.
I invite you to read this chapter in the documentation: "Failing Over Nodes"