Client fails to reconnect to server of non-default memcached bucket after failover and add back

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Duplicate
Affects Version/s: 1.0.3
Fix Version/s: 1.1.2
Component/s: library
Security Level: Public
Labels:
- customer

Description

In earlier tests with reconnecting to a node on failover we used default memcached bucket. But when we tested the same scenario with a non-default bucket, we noticed the client did not reconnect (due to a null pointer exception internally). I have attached the SDK logs for this scenario where we used "IndexByLniataData" memcached bucket. The problem presents when adding the node back after a failover.

11:34:43,411 DEBUG [Memcached IO over {MemcachedConnection to /10.14.5.119:11210}] [CouchbaseMemcachedConnection] Selecting with delay of 3038ms
Exception in thread "Thread-3" java.lang.NullPointerException
        at net.spy.memcached.auth.AuthThread.buildOperation(AuthThread.java:117)
        at net.spy.memcached.auth.AuthThread.run(AuthThread.java:86)

Logs/stack trace attached.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

sc_1_plabq11.dev.sabre.com.out.gz

28/Jun/12 9:56 AM

1.62 MB

Perry Krug

Issue Links

depends on

SPY-102 Ensure all nodes are in the list of nodes after reconfiguration

is duplicated by

SPY-111 Assertion/NPE Exception when trying to use auth and failover.

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Matt Ingenthron added a comment - 22/Aug/12 10:22 PM

I've spent a bit of time analyzing this issue, and it's not clear what the cause is. It is correct though that this would cause the auth thread to die, and as such authentication to the node would never complete.

There is a safeguard already in that the continuous timeout threshold will kick in and then the connection will be rebuilt. I don't know if this issue comes up all of the time, but assuming it's a rare event we'd see 1000 operations timeout (by default) followed by the connection being rebuilt.

We'd have to add some diagnostic information to the client and reliably reproduce this to identify the issue. I think the scenario is:
1) set up a cluster of say 3 nodes
2) configure a client, have it work with an authenticated memcached bucket on the cluster
3) faillover a node by clicking on "failover" in the console
4) add the node back by clicking on "add back"

Is this correct?

Show

Matt Ingenthron added a comment - 22/Aug/12 10:22 PM I've spent a bit of time analyzing this issue, and it's not clear what the cause is. It is correct though that this would cause the auth thread to die, and as such authentication to the node would never complete. There is a safeguard already in that the continuous timeout threshold will kick in and then the connection will be rebuilt. I don't know if this issue comes up all of the time, but assuming it's a rare event we'd see 1000 operations timeout (by default) followed by the connection being rebuilt. We'd have to add some diagnostic information to the client and reliably reproduce this to identify the issue. I think the scenario is: 1) set up a cluster of say 3 nodes 2) configure a client, have it work with an authenticated memcached bucket on the cluster 3) faillover a node by clicking on "failover" in the console 4) add the node back by clicking on "add back" Is this correct?

Hide

Permalink

Perry Krug added a comment - 23/Aug/12 3:09 AM

That appears correct. The customer has been able to reliably reproduce this, but since so much time has passed I would be hesitant in going back to them if not necessary...

Show

Perry Krug added a comment - 23/Aug/12 3:09 AM That appears correct. The customer has been able to reliably reproduce this, but since so much time has passed I would be hesitant in going back to them if not necessary...

Raghavan Srinivas made changes - 30/Aug/12 7:57 AM