Couchbase Java Client

Client fails to reconnect to server of non-default memcached bucket after failover and add back

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Duplicate
  • Affects Version/s: 1.0.3
  • Fix Version/s: 1.1.2
  • Component/s: library
  • Security Level: Public
  • Labels:

Description

In earlier tests with reconnecting to a node on failover we used default memcached bucket. But when we tested the same scenario with a non-default bucket, we noticed the client did not reconnect (due to a null pointer exception internally). I have attached the SDK logs for this scenario where we used "IndexByLniataData" memcached bucket. The problem presents when adding the node back after a failover.
  
11:34:43,411 DEBUG [Memcached IO over {MemcachedConnection to /10.14.5.119:11210}] [CouchbaseMemcachedConnection] Selecting with delay of 3038ms
Exception in thread "Thread-3" java.lang.NullPointerException
        at net.spy.memcached.auth.AuthThread.buildOperation(AuthThread.java:117)
        at net.spy.memcached.auth.AuthThread.run(AuthThread.java:86)

Logs/stack trace attached.

Issue Links

depends on

Bug - A problem which impairs or prevents the functions of the product. SPY-102 Ensure all nodes are in the list of nodes after reconfiguration

  • Major - Major loss of function.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.
is duplicated by

Bug - A problem which impairs or prevents the functions of the product. SPY-111 Assertion/NPE Exception when trying to use auth and failover.

  • Major - Major loss of function.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Ingenthron added a comment -
I've spent a bit of time analyzing this issue, and it's not clear what the cause is. It is correct though that this would cause the auth thread to die, and as such authentication to the node would never complete.

There is a safeguard already in that the continuous timeout threshold will kick in and then the connection will be rebuilt. I don't know if this issue comes up all of the time, but assuming it's a rare event we'd see 1000 operations timeout (by default) followed by the connection being rebuilt.

We'd have to add some diagnostic information to the client and reliably reproduce this to identify the issue. I think the scenario is:
1) set up a cluster of say 3 nodes
2) configure a client, have it work with an authenticated memcached bucket on the cluster
3) faillover a node by clicking on "failover" in the console
4) add the node back by clicking on "add back"

Is this correct?
Show
Matt Ingenthron added a comment - I've spent a bit of time analyzing this issue, and it's not clear what the cause is. It is correct though that this would cause the auth thread to die, and as such authentication to the node would never complete. There is a safeguard already in that the continuous timeout threshold will kick in and then the connection will be rebuilt. I don't know if this issue comes up all of the time, but assuming it's a rare event we'd see 1000 operations timeout (by default) followed by the connection being rebuilt. We'd have to add some diagnostic information to the client and reliably reproduce this to identify the issue. I think the scenario is: 1) set up a cluster of say 3 nodes 2) configure a client, have it work with an authenticated memcached bucket on the cluster 3) faillover a node by clicking on "failover" in the console 4) add the node back by clicking on "add back" Is this correct?
Hide
Permalink
Perry Krug added a comment -
That appears correct. The customer has been able to reliably reproduce this, but since so much time has passed I would be hesitant in going back to them if not necessary...
Show
Perry Krug added a comment - That appears correct. The customer has been able to reliably reproduce this, but since so much time has passed I would be hesitant in going back to them if not necessary...
Hide
Permalink
Matt Ingenthron added a comment -
There is an open changeset for this. Please determine if it is correct, needs to go in.
Show
Matt Ingenthron added a comment - There is an open changeset for this. Please determine if it is correct, needs to go in.
Hide
Permalink
Michael Nitschinger added a comment -
Duplicate of Spy-111
Show
Michael Nitschinger added a comment - Duplicate of Spy-111

People

  • Assignee:
    Michael Nitschinger
    Reporter:
    Perry Krug
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: