[JCBC-272] removing non-bootstrap node with memcached bucket okay, but adding the same node results in failures Created: 13/Mar/13  Updated: 15/Mar/13  Resolved: 15/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.5
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Client 1.1.2 with required spymemcached 2.8.11. Server 2.0.0 with a single authenticated memcached bucket. Assertions enabled.

Issue Links:
Duplicate
duplicates JCBC-271 Adding a node to an existing cluster ... Resolved

 Description   
From the attached log, one will see first the removal of a node (click remove -> click rebalance), and things are generally okay.

Following that, you'll see the node added (via the config wizard, joining to node 192.168.1.200), and one configuration received, but nothing actually used yet.

Following that, you'll see the node rebalanced in with a set of assertion errors and then workload ceases.

Notably:
2013-03-13 19:29:03.924 INFO com.couchbase.client.CouchbaseMemcachedConnection: Reconnecting {QA sa=/192.168.1.201:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 139891 Key: pool-25-thread-1:48454, topWop=null, toWrite=0, interested=1}
Exception in thread "Memcached IO over {MemcachedConnection to /192.168.1.200:11210 /192.168.1.202:11210 /192.168.1.201:11210}" java.lang.AssertionError: Attempting to overwrite channel

 Comments   
Comment by Deepti Dawar [ 14/Mar/13 ]
In this scenario, I am encountering something like this :

[INFO 33.11 cbsdk.scenario failover.py:198] Sleeping for 60 seconds after failover
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.MemcachedConnection handleIO
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to exception on {QA sa=/10.3.2.57:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149, topWop=null, toWrite=0, interested=1}
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] java.io.IOException: Disconnected unexpected, will reconnect.
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:526)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:454)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.MemcachedConnection queueReconnect
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.2.57:11210, #Rops=1, #Wops=0, #iq=1, topRop=Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149, topWop=null, toWrite=0, interested=1}, attempt 0.
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] java.lang.RuntimeException: Cancelled
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:169)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:132)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:64)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
[SDKD(WARNING) 33.16 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM com.couchbase.client.CouchbaseMemcachedConnection reconfigure
[SDKD(WARNING) 33.17 cbsdk.sdkd.local executor.py:66] INFO: Scheduling Node /10.3.2.57:11210for shutdown.
Generated at Wed Apr 23 07:45:11 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.