Java Client Connection keep closing and reopening, Auth failure

Java Client: 1.1.9
Couchbase Server: 2.1.1 Enterprise
OS: CentOS in VirtualBox

int numOfCouchbaseConnection = 1
List<URI> hosts = Arrays.asList(new URI("http://127.0.0.1:8091/pools"));
 
    CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
    builder.setProtocol(ConnectionFactoryBuilder.Protocol.BINARY);
    builder.setShouldOptimize(true);
    builder.setOpTimeout(32);
 
    couchbaseClients = new ArrayList<>();
    for (int i = 0; i < numOfCouchbaseConnection; i++) {
      couchbaseClients.add(new CouchbaseClient(builder.buildCouchbaseConnection(hosts, "products", "Administrator", "")));
    }

If I switch the Couchbase Server to 2.0.1 Community, there will be another error

2013-09-05 12:38:56.932 WARN net.spy.memcached.auth.AuthThread:  Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:56.938 ERROR net.spy.memcached.protocol.binary.SASLStepOperationImpl:  Error:  Auth failure
2013-09-05 12:38:56.939 INFO com.couchbase.client.CouchbaseConnection:  Reconnection due to exception handling a memcached operation on {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL steps operation, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure.
OperationException: SERVER: Auth failure
 at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:164)
 at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:176)
 at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.finishedPayload(SASLBaseOperationImpl.java:93)
 at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:162)
 at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.readFromBuffer(SASLBaseOperationImpl.java:41)
 at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:563)
 at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:480)
 at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:261)
 at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
2013-09-05 12:38:56.940 WARN com.couchbase.client.CouchbaseConnection:  Closing, and reopening {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL steps operation, topWop=null, toWrite=0, interested=1}, attempt 0.
2013-09-05 12:38:56.940 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl:  Discarding partially completed op: SASL steps operation
2013-09-05 12:38:57.039 WARN net.spy.memcached.auth.AuthThread:  Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:58.941 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=0}
2013-09-05 12:38:58.942 INFO com.couchbase.client.CouchbaseConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@7fe431c2
2013-09-05 12:38:58.943 WARN net.spy.memcached.auth.AuthThreadMonitor:  Incomplete authentication interrupted for node {QA sa=10.100.20.250/10.100.20.250:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=8}
2013-09-05 12:38:58.944 WARN net.spy.memcached.auth.AuthThread:  Authentication failed to 10.100.20.250/10.100.20.250:11210
2013-09-05 12:38:58.948 INFO net.spy.memcached.auth.AuthThread:  Authenticated to 10.100.20.250/10.100.20.250:11210

I have the same issue. I spend stress testing and at the same time kill one of nodes. There are no operations I have "WARN net.spy.memcached.auth.AuthThread: Authentication failed to ..." What mechanism will allow to continue the work of the application after the restoration of failing node.

1 Answer

« Back to question.

Hello,

When you connect to Couchbase cluster you should not use the Administrator user. The Administrator should only be used to do some administration operation such as create/edit bucket using the ClusterManager class

When you connect to the cluster to do "normal operations" you have 2 options
- connect without any security/password
- connect with an SASL password (that you set at the bucket level).

So for example in your case you will set a password to the products bucket, and connect using:

CouchbaseClient cb = new CouchbaseClient(builder.buildCouchbaseConnection(hosts, "products", "mypassword"));

Also I see in the code snippet you put in the question that you are creating a list of CouchbaseClient. This is not a a good idea. You just need one client in your application. The "pool" is manage internally in the client.

Regards
Tug
@tgrall

Hey tgrall! I did a performance test for my pool

#threads	#connections	#queries
1	        1             21869
2         1             22191
4         1             21212
2         2             40548
4         4             103663
1         2             23193
1         4             20889

My Testing Code

//Couchbase.java
...
System.out.println(System.currentTimeMillis());
    couchbaseClients = new ArrayList<>();
    for (int i = 0; i < numOfCouchbaseConnection; i++) {
      CouchbaseClient couchbaseClient = new CouchbaseClient(hosts, "products", "");
      couchbaseClients.add(new AbstractMap.SimpleEntry<>(couchbaseClient, couchbaseClient.getView("products", "default")));
    }
 
		for (int i = 0; i < numOfCurrency; i++)
		{
			Thread thread = new QueryThread(categories.get(random.nextInt(categories.size())));
			threads.add(thread);
		}
 
    for (int i = 0; i < numOfCurrency; i++)
		{
			threads.get(i).start();
		}
public synchronized static Map.Entry<CouchbaseClient, View> getCouchbaseClient() {
    couchbaseIndex = couchbaseIndex % couchbaseClients.size();
    return couchbaseClients.get(couchbaseIndex++);
  }
...
//
 
//QueryThread.java
...
public void run()
	{
    System.out.println(System.currentTimeMillis());
		long start = System.currentTimeMillis();
		long end = start + Couchbase.seconds;
		while(System.currentTimeMillis() < end)
		{
      Map.Entry<CouchbaseClient, View> a = Couchbase.getCouchbaseClient();
      Query query = new Query();
			query.setStale(Stale.OK);
			query.setRange(ComplexKey.of("waejk;jkgrad", "3789560789", m_category, null), ComplexKey.of("waejk;jkgrad", "3789560789", m_category, ""));
			ViewResponse viewResponse = a.getKey().query(a.getValue(), query);
			Couchbase.atomicInteger.getAndIncrement();
		}
		System.out.println(System.currentTimeMillis());
		System.out.println(Couchbase.atomicInteger.getAndIncrement());
	}

No matter how many threads share the same connection, the number of queries of different threads are nearly same. if you have multiple connections for different threads, there will be a huge difference in performance

No matter if you use auth or you don't, authentication failure after one of servers restart/crash.
If only one thread is making requests through the client (single instance in my case, not a list of) then occasionally auth failure disappears (after ~5-10 minutes), if many threads try to do operations on the client (many means 50-100), reconnection time increases or even no reconnection occurs.

Any thoughts?