Implement Replica Read Capability (NCBC-493)

[NCBC-568] Update IVBucket to support more than one replica Created: 29/Jul/14  Updated: 29/Jul/14  Resolved: 29/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Technical task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Jeff Morris [ 29/Jul/14 ]
http://review.couchbase.org/#/c/39991/




[NCBC-479] Add Retry logic for NMV's Created: 14/May/14  Updated: 29/Jul/14  Resolved: 29/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0.0
Fix Version/s: 2.0 Beta

Type: Task Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Jeff Morris [ 29/Jul/14 ]
CR: http://review.couchbase.org/#/c/39989/




[NCBC-550] Thread A is disposing of Bucket while thread B is still using it Created: 11/Jul/14  Updated: 28/Jul/14  Resolved: 27/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 DP3

Type: Bug Priority: Test Blocker
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File log_25.txt    
Issue Links:
Relates to
relates to NCBC-545 couchApiBase in pools/default/buckets... Resolved

 Description   

2014-07-11 10:56:35,176 [14] INFO Couchbase.IO.Operations.OperationResult`1 - Received config rev#1073
2014-07-11 10:56:35,176 [14] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1073
2014-07-11 10:56:35,176 [14] WARN Couchbase.Configuration.Server.Providers.ConfigProviderBase - A ConfigObserver for the bucket default was not found. Provider has been disposed: True
2014-07-11 10:56:39,401 [14] INFO Couchbase.Core.Buckets.CouchbaseBucket - Requires retry SimpleKeyREP73REP73REP73REP73REP73REP73REP73

 Comments   
Comment by Jeff Morris [ 13/Jul/14 ]
commit 28ea4d12d13803a472b2e31c848c1aa47f05ad67

This should fix the threading issues discovered thus far. Importantly it adds ref counting to bucket classes, so that the dispose will only be called by the final thread using the shared bucket.

Let's try another set of situational tests and see what happens.
Comment by Jeff Morris [ 13/Jul/14 ]
Note: use this path (upper case "Src"): couchbase-csharp-client-ng/Src/ _not_ couchbase-csharp-client-ng/src/

Comment by Andrei Baranouski [ 14/Jul/14 ]
Hi Jeff,

seems like it's still reproduced: http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/25/console


2014-07-14 01:09:51,763 [14] INFO Couchbase.IO.Operations.OperationResult`1 - Received config rev#1163
2014-07-14 01:09:51,763 [14] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1163
2014-07-14 01:09:51,767 [14] WARN Couchbase.Configuration.Server.Providers.ConfigProviderBase - A ConfigObserver for the bucket default was not found. Provider has been disposed: True
2014-07-14 01:09:51,767 [14] INFO Couchbase.Core.Buckets.CouchbaseBucket - Requires retry SimpleKeyREP0REP0REP0REP0REP0REP0REP0REP0


client's logs attached


Comment by Andrei Baranouski [ 14/Jul/14 ]
client's logs with 2.0 commit 28ea4d against 3.0.--918
Comment by Jeff Morris [ 14/Jul/14 ]
Andrei -

Looks like a different error:

[51.36 INFO] (SDKD log:137)  at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Couchbase.Core.Server.GetBaseViewUri(String bucketName) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Server.cs:line
   at Couchbase.Core.Server.Send[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Server.cs:line 85
   at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 492
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 122
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 225
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 191
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 123
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

If it's a different error, then we should open a new ticket. Let me know if this is the case.

-Jeff

Comment by Jeff Morris [ 14/Jul/14 ]
Quick question: in the situational test are you disposing the bucket? I am going to add more tracing and then provide another SHA1 for testing.
Comment by Andrei Baranouski [ 14/Jul/14 ]
Hi Jeff,

yes, stack trace in the test is different but client's logs are the same as in the ticket, so if it's the new issue...

currently in my runs I don't use the bucket disposing because we re-use client instances in all threads/phases in the test/scenario https://github.com/couchbase/sdkd-net/blob/client-2.0/src/Sdkd/Couchbase/ClientFactory.cs#L33

but i will add at the end of the tests or when we run with ShareClients = false


Comment by Jeff Morris [ 14/Jul/14 ]
>>WARN Couchbase.Configuration.Server.Providers.ConfigProviderBase - A ConfigObserver for the bucket default was not found. Provider has been disposed: True

The last sentence on that line makes me think that somehow the bucket instance is being disposed either implicitly (via finalizer) or explicitly.

Sure enough >> DEBUG Couchbase.Core.ClusterManager - Disposing!!!

The patch in place should keep disposed from being called until the last instance has finished using it. Hmm.
Comment by Jeff Morris [ 23/Jul/14 ]
This should be resolved as long as the bucket instance is not disposed or reaches finalization (in other words it's the applications responsibility to ensure that the bucket isn't disposed). If an operation is attempted on a disposed bucket, an ObjectDisposedException will be thrown which must be handled by the app.
Comment by Don Pinto [ 25/Jul/14 ]
Jeff, Can you please sync-up with Andrei over Skype. This is still looking as a blocker from his side.

Thanks,
Comment by Jeff Morris [ 27/Jul/14 ]
http://review.couchbase.org/#/c/39935/
Comment by Jeff Morris [ 28/Jul/14 ]
SHA for testing: commit 9ce8806edc38503086af1fe9af66484b5766859d




[NCBC-547] ExecuteIncrement requires value to be stored as a numeric string. Created: 09/Jul/14  Updated: 28/Jul/14  Resolved: 28/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: docs, library
Affects Version/s: 1.3.7
Fix Version/s: 1.3.8

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to

 Description   
This may be handled in the documentation instead of as a code patch.

 Comments   
Comment by Jeff Morris [ 09/Jul/14 ]
The doucmentation here: http://docs.couchbase.com/couchbase-sdk-net-1.3/index.html#increment-methods

"...the Increment methods may not be used with keys that were first created with a Store method. To initialize a counter, you must first use either Increment or Increment with a default value."

Should be something like:

"...the Increment methods may be used with keys that were first created with a Store method, if and only if the value is a numeric string. To initialize a counter, you must first use either Increment or Increment with a default value."

In general though, the correct way to do this is via Increment/Decrement _not_ by incrementing a key stored with a Store method.
Comment by Jeff Morris [ 28/Jul/14 ]
This is expected behavior from the perspective of the server. In general, if you are going to use Incr/Decr, don't use Store() to create the keys.




[NCBC-563] NullReferenceException in CouchbaseConfigContext.LoadConfig Created: 25/Jul/14  Updated: 25/Jul/14  Resolved: 25/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0.0

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive log.zip    

 Description   
very often in different tests I see the following exception


http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/38/consoleFull

Running SvcRestart-HYBRID: 2014-07-25 05:20:15
..............

[365.86 INFO] (SDKD log:137) Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
[366.20 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[366.20 INFO] (SDKD log:137)  at Couchbase.Configuration.CouchbaseConfigContext.LoadConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\CouchbaseConfigContext.cs:line 51
   at Couchbase.Configuration.ConfigContextBase..ctor(IBucketConfig bucketConfig, ClientConfiguration clientConfig, Func`2 ioStrategyFactory, Func`3 connectionPoolFactory, Func`5 saslFactory, IByteConverter converter, ITypeSerializer serializer) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 51
   at Couchbase.Configuration.CouchbaseConfigContext..ctor(IBucketConfig bucketConfig, ClientConfiguration clientConfig, Func`2 ioStrategyFactory, Func`3 connectionPoolFactory, Func`5 saslFactory, IByteConverter converter, ITypeSerializer serializer) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\CouchbaseConfigContext.cs:line 22
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 63
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 44
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 155
   at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 82
   at Sdkd.Couchbase.ClientFactory.FindExistingClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 55
   at Sdkd.Couchbase.ClientFactory.CreateClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 262
   at SdkdConsole.Daemon.Handle.CreateHandle(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 84
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 103
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[366.20 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
[366.21 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@203.259
[366.21 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]


client logs http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/38/artifact/log.txt

Jeff, believe that this is a problem on client side?



 Comments   
Comment by Jeff Morris [ 25/Jul/14 ]
Yes, I'll take a look into this today Andrei.
Comment by Jeff Morris [ 25/Jul/14 ]
Andrei - can you attach the SDK logs to this ticket?
Comment by Jeff Morris [ 25/Jul/14 ]
This is what's happening:

When the client attempts to bootstrap and the node it elects to bootstrap off of is in an inconsistent state, during the SASL process the first fails and thus the SaslMechanism is never chosen leading to a subsequent NRE:

https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Authentication/SASL/SaslFactory.cs#L24

https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Configuration/CouchbaseConfigContext.cs#L51

The simple solution seems to be check for null - which will push the problem further out. It should be a "cannot bootstrap error, please try again" perhaps looping through the bootstrap list in the configuration.
Comment by Jeff Morris [ 25/Jul/14 ]
commit b5fc255555f26eb5f691eb0441c7997577fc79e6




[NCBC-504] Use ForwardVBucketMap when available Created: 06/Jun/14  Updated: 24/Jul/14  Resolved: 24/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 Beta

Type: Task Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Jeff Morris [ 24/Jul/14 ]
commit 8b030587562a1d21770505f2edc60a69f353faab




[NCBC-552] Ensure that bucket throws ObjectDisposedExceptions if an operation is attempted after calling Dispose() Created: 14/Jul/14  Updated: 24/Jul/14  Resolved: 24/Jul/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 DP3

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When a thread calls Dispose on a bucket and no other thread is referencing it, subsequent operations will fail with an ObjectDisposedException: "This bucket [{0}] has been disposed! Performing operations on a disposed bucket is not supported!"

 Comments   
Comment by Jeff Morris [ 24/Jul/14 ]
http://review.couchbase.org/#/c/39381/




Generated at Thu Jul 31 04:46:09 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.