[JCBC-148] Issue with Observe API Persist.TWO and 1 dead node: Time Out when doing set operation

Mon, 3 Dec 2012 03:37:02 -0600

I have a very simple Java program that connect to the 2 nodes and do a set with the following code:

1. So I try to connect to multiple nodes
{code}
            List<URI> couchbaseServerUris = new ArrayList<URI>();
            couchbaseServerUris.add( new URI("http://192.168.0.108:8091/pools") );
            couchbaseServerUris.add( new URI("http://192.168.0.104:8091/pools") );
            CouchbaseClient client = new CouchbaseClient( couchbaseServerUris , "default" , "" );
{code}

2. Then I call the set operation
{code}

        OperationFuture<Boolean> stored = client.set( "my-dummy-key",0, "{\"name\" : \"foo\", \"title\" : \"bar-test\"}", PersistTo.TWO);
{code}

---
So everything is working as expected when the 2 nodes are up.

When I kill 1 node (for example : disconnecting, or stopping, or pausing the Ubuntu VM) I have the following behavior:

When I execute this program:
1- I have an exception saying that 1 node is down : Expected behavior (even if we could avoid a long stack trace)
{code}
2012-11-18 08:14:55.830 WARN com.couchbase.client.vbucket.ConfigurationProviderHTTP: Connection problems with URI http://192.168.0.108:8091/pools ...skipping
java.net.ConnectException: Host is down
{code}

2- When I do the set the program is stopped/blocked until it reaches a network timeout
2012-11-18 08:20:13.462 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
{code}
Error while storing : Observe Timeout - Polled Unsuccessfully for at least 40 seconds.
2012-11-18 08:20:13.466 INFO done : true
done : {OperationStatus success=false: Observe Timeout - Polled Unsuccessfully for at least 40 seconds.}
com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
2012-11-18 08:20:13.467 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
{code}

Note that it is only happening with PersistTo.TWO
if I use PersistTo.MASTER or PersistTo.ONE : the program is executed with no error and no stop
if I use PersistTo.THREE ( or more) : the program is executed, no stop with the expected observe message : ( Error while storing : Requested persistence to 3 node(s), but only 2 are available.
)