Couchbase Situational Behaviors

h1. Situational Behaviors

In order to properly write a program against a system, especially a complex system such as a distributed database, it is necessary to understand how that system behaves in situations that are outside normal operation.  This document will try to describe the different situations and what application code can expect to need to handle in those situations.
{info}This document is still under development and review. Comments and questions are greatly appreciated.{info}

h2. Maintenance Operations


h3. When a Node is Added

During a node addition, all data operations and query operations will continue to operate normally.  Application code should not be able to discern (other than effects to resource utilization) that anything is happening.

h3. When a Node is Removed

When a node is removed, all data operations and query operations will continue to operate normally.  Again a{color:#333333}pplication code should not be able to discern (other than effects to resource utilization) that anything is happening.{color}


{note}{color:#333333}There is a known issue with this behavior when using server releases prior to 2.0. With 1.6, 1.7 and 1.8 an application will see errors as the end of the rebalance completes. These will not be retried by the client. This condition is also, unfortunately, not distinguishable from a node failure.{color}

{color:#333333}The application code may retry this operation shortly after the rebalance completes and it will most likely be handled normally.{color}

h3. When a Node is Added and Removed

In this scenario, data is migrating from the node being removed to the node being added.  Frequently this is referred to as a _swap rebalance._  

In  

h2. Node Failures


h3. When a Node Fails

The exact behavior depends on whether the operation is related to a data update or modification, or a query.

If the the requested operation is a data operation, such as a retrieving a document with a get() or storing a document with a set(), the success or failure of the operation would be based on which node it is supposed to be for.  In Couchbase Server, this is determined by a hashing algorithm driven by the key/id for the given document.  For example, "foo" could be mapped to the hostname _node1.example.com_ and "bar" could be mapped to _node2.example.com_.  In the event the server for _node1_ fails, any attempt to modify the key/id "foo" will eventually fail.  It will not necessarily immediately fail, as the client library will continue to try to reconnect to the node.  As long as the operation requested is still valid and has not reached the timeout threshold, the client will continue to try to service the request.

If the requested operation is a query, the request will be routed to an available node.  The JSON response from the cluster will contain an errors\[\] array, and the client may indicate an error condition.  Through modification of the on_error parameter to the query, the client may request different behavior from the server.  Note that this means a client *may get query results which are incomplete* and it is the responsibility of the application to handle these error conditions.  See the Couchbase Server documentation for more info.  

h3. When All Nodes Fail

When all nodes fail, no operations will be available.  If the client had started and been working with a cluster previously, it would continue to try to connect to that cluster until it becomes available or the client is terminated.

If a newly instantiated client tries to connect to the cluster, it will fail to initialize.

h2. Node Failover


h3. When Failover is Triggered Without Full Failure


h3. When Autofailover is Triggered Because a Node is Down


h3. When Failover is Triggered with Full Failure


h3. When a Node is Added Back and Rebalanced In


h2. Noteable Circumstances

{color:#000000}{*}When the Bootstrap Node for a Client Fails{*}{color}

In this case, there are two differ