[NCBC-645] [ssl] test Rb2Out-HYBRID stuck with 'Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException' in client logs Created: 17/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-645.zip     Text File test_logs.txt    

 Description   
c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
ig.ini -I sdkd.args --variants HYBRID -d all:debug
[3.21 INFO] (BRun run:436) Initializing S3
[7.80 INFO] (BRun run:441) Initializing history database

============================================================
Running Rb2Out-HYBRID: 2014-09-17 02:29:58
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\Rb2Out-HYBRID\09-17-14\0341
98\28852a8b5890fe94895277f14f7f4f66

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase
        --cluster_useSSL=True

[447.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[448.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[449.93 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[449.94 INFO] (Scenario run:97) CHANGE phase done
[449.95 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[539.95 INFO] (Scenario run:105) REBOUND done. Will collect results
[539.95 DEBUG] (Handle sendMessageAsync:183) > CANCEL@103.72
[539.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 103
[539.97 DEBUG] (Handle receiveMessage:158) < CANCEL@103.72
[539.97 DEBUG] (Handle sendMessageAsync:183) > CANCEL@104.72
[539.98 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 104
[2705.74 DEBUG] (Handle receiveMessage:158) < CANCEL@104.72
[2706.66 DEBUG] (Handle sendMessageAsync:183) > CANCEL@105.72
[2707.19 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 105
Terminate batch job (Y/N)? Y




part of the client logs when test stuck(02:37 - 03:14)

2014-09-17 02:37:18,228 [7] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 7
2014-09-17 02:37:18,227 [47] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 40d8fbd1-7b5d-4334-a688-065d2575cca6 on 10.3.3.206:11207 - a17e5df5-7c63-4030-ba02-41309e68ea55
2014-09-17 02:37:18,225 [21] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: abf3e988-cbfc-4795-8523-325094b9e719 on 10.3.121.134:11207 - 624b4df1-35dc-4842-8a23-7db2310b001b
2014-09-17 02:37:18,232 [50] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#808 on thread 50
2014-09-17 02:37:18,230 [9] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 9
2014-09-17 02:37:18,231 [6] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#820 on thread 6
2014-09-17 02:37:18,233 [43] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#828 on thread 43
2014-09-17 02:37:18,253 [18] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#840 on thread 18
2014-09-17 02:37:18,252 [28] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 28
2014-09-17 02:37:18,254 [53] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#798 on thread 53
2014-09-17 02:37:18,258 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 42
2014-09-17 02:37:18,258 [46] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 981 for key SimpleKeyREP5537REP5537REP5537REP5537REP5537 - rev798
2014-09-17 02:37:18,210 [49] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 515 for key SimpleKeyREP5382REP5382REP5382REP5382REP5382 - rev870
2014-09-17 02:37:19,297 [59] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [13] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [62] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [20] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [12] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [15] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,301 [61] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [60] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,300 [11] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:51,391 [7] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 427 for key SimpleKeyREP5352REP5352REP5352REP5352REP5352 - rev870
2014-09-17 03:14:51,391 [30] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 226 for key SimpleKeyREP5697REP5697REP5697REP5697REP5697 - rev870
2014-09-17 03:14:51,391 [24] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 958 for key SimpleKeyREP5964REP5964REP5964REP5964REP5964 - rev817
2014-09-17 03:14:51,391 [48] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 456 for key SimpleKeyREP5799REP5799REP5799REP5799REP5799 - rev810
2014-09-17 03:14:51,391 [8] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 461 for key SimpleKeyREP6028REP6028REP6028REP6028REP6028 - rev799
2014-09-17 03:14:51,334 [49] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5382REP5382REP5382REP5382REP5382 using server 10.3.121.134:11207
2014-09-17 03:14:51,391 [44] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 246 for key SimpleKeyREP5854REP5854REP5854REP5854REP5854 - rev870
2014-09-17 03:14:51,391 [6] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 34 for key SimpleKeyREP5909REP5909REP5909REP5909REP5909 - rev820
2014-09-17 03:14:51,391 [9] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 728 for key SimpleKeyREP5752REP5752REP5752REP5752REP5752 - rev800
2014-09-17 03:14:51,391 [43] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 554 for key SimpleKeyREP5750REP5750REP5750REP5750REP5750 - rev828
2014-09-17 03:14:51,391 [50] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 109 for key SimpleKeyREP5503REP5503REP5503REP5503REP5503 - rev808
2014-09-17 03:14:51,391 [18] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 633 for key SimpleKeyREP5670REP5670REP5670REP5670REP5670 - rev840
2014-09-17 03:14:51,391 [28] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 825 for key SimpleKeyREP5706REP5706REP5706REP5706REP5706 - rev800
2014-09-17 03:14:51,391 [53] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 964 for key SimpleKeyREP5289REP5289REP5289REP5289REP5289 - rev798
2014-09-17 03:14:51,391 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 610 for key SimpleKeyREP5850REP5850REP5850REP5850REP5850 - rev870
2014-09-17 03:14:51,334 [46] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5537REP5537REP5537REP5537REP5537 using server 10.3.3.206:11207
2014-09-17 03:14:51,631 [45] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5291REP5291REP5291REP5291REP5291
2014-09-17 03:14:51,631 [47] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5660REP5660REP5660REP5660REP5660
2014-09-17 03:14:51,631 [21] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5520REP5520REP5520REP5520REP5520
2014-09-17 03:14:51,632 [52] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5089REP5089REP5089REP5089REP5089
2014-09-17 02:37:19,300 [63] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:54,754 [7] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5352REP5352REP5352REP5352REP5352 using server 10.3.3.206:11207
2014-09-17 03:14:54,756 [49] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: c1a7c0f9-9ae6-406a-870f-221d1cd5abc7 | 10.3.121.134:11207 | [0, 2] - 45e4bde8-afd4-4084-b4fe-1952631c5f47 - Disposed: False
2014-09-17 03:14:54,760 [18] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5670REP5670REP5670REP5670REP5670 using server 10.3.3.206:11207
2014-09-17 03:14:54,760 [28] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5706REP5706REP5706REP5706REP5706 using server 10.3.3.206:11207
2014-09-17 03:14:54,762 [46] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: dfdafeeb-7f31-4104-94ea-a7c9c951aa6e | 10.3.3.206:11207 | [0, 2] - 728e6fe1-8778-4acd-b15d-832d832b0a1a - Disposed: False
2014-09-17 03:14:54,760 [53] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5289REP5289REP5289REP5289REP5289 using server 10.3.3.206:11207
2014-09-17 03:14:54,761 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5850REP5850REP5850REP5850REP5850 using server 10.3.3.206:11207
2014-09-17 03:14:54,749 [20] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,750 [15] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [59] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,747 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,748 [62] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [50] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5503REP5503REP5503REP5503REP5503 using server 10.3.121.134:11207
2014-09-17 03:14:54,750 [12] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,758 [9] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5752REP5752REP5752REP5752REP5752 using server 10.3.3.206:11207
2014-09-17 03:14:54,754 [30] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5697REP5697REP5697REP5697REP5697 using server 10.3.121.134:11207
2014-09-17 03:14:54,752 [60] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,753 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,755 [48] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5799REP5799REP5799REP5799REP5799 using server 10.3.3.206:11207



 Comments   
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you define "stuck"? Seeing exceptions in the logs is expected, but if the client has stopped completely, this is definitely a blocking issue.

-Jeff
Comment by Andrei Baranouski [ 17/Sep/14 ]
yes, the client has stopped completely
 I only got a new batch of activities (in logs), after I click Ctrl+c in terminal where I run the tests.
That's all I can tell you now. let me know if you need some detail on this issue
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you do a "Debug->Break All" and look at the threads to see where it's blocking? I am guessing it's on one of the Wait()'s that doesn't have a timeout set.

Thanks,

Jeff
Comment by Jeff Morris [ 29/Sep/14 ]
I doubt it's related and I may create an MB, but view performance is erratic (~90ms to ~15s) during a node removal/fail-over and subsequent rebalance:

Line 338: 2014-09-29 19:12:39,724 [Runner thread] DEBUG Wintellect.OperationTimer2 - 330 (GCs= 2)
Line 340: 2014-09-29 19:12:40,134 [Runner thread] DEBUG Wintellect.OperationTimer2 - 376 (GCs= 2)
Line 342: 2014-09-29 19:12:40,484 [Runner thread] DEBUG Wintellect.OperationTimer2 - 326 (GCs= 2)
Line 344: 2014-09-29 19:12:40,860 [Runner thread] DEBUG Wintellect.OperationTimer2 - 344 (GCs= 2)
Line 346: 2014-09-29 19:12:41,297 [Runner thread] DEBUG Wintellect.OperationTimer2 - 408 (GCs= 2)
Line 348: 2014-09-29 19:12:41,656 [Runner thread] DEBUG Wintellect.OperationTimer2 - 325 (GCs= 2)
Line 350: 2014-09-29 19:12:42,070 [Runner thread] DEBUG Wintellect.OperationTimer2 - 380 (GCs= 2)
Line 352: 2014-09-29 19:12:42,519 [Runner thread] DEBUG Wintellect.OperationTimer2 - 422 (GCs= 2)
Line 354: 2014-09-29 19:12:43,112 [Runner thread] DEBUG Wintellect.OperationTimer2 - 558 (GCs= 2)
Line 356: 2014-09-29 19:12:43,767 [Runner thread] DEBUG Wintellect.OperationTimer2 - 620 (GCs= 2)
Line 358: 2014-09-29 19:12:44,311 [Runner thread] DEBUG Wintellect.OperationTimer2 - 514 (GCs= 2)
Line 360: 2014-09-29 19:12:44,889 [Runner thread] DEBUG Wintellect.OperationTimer2 - 550 (GCs= 2)
Line 362: 2014-09-29 19:12:45,555 [Runner thread] DEBUG Wintellect.OperationTimer2 - 620 (GCs= 2)
Line 364: 2014-09-29 19:12:46,457 [Runner thread] DEBUG Wintellect.OperationTimer2 - 855 (GCs= 2)
Line 366: 2014-09-29 19:12:47,389 [Runner thread] DEBUG Wintellect.OperationTimer2 - 878 (GCs= 2)
Line 368: 2014-09-29 19:12:53,148 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,712 (GCs= 2)
Line 370: 2014-09-29 19:12:53,917 [Runner thread] DEBUG Wintellect.OperationTimer2 - 739 (GCs= 2)
Line 372: 2014-09-29 19:12:54,685 [Runner thread] DEBUG Wintellect.OperationTimer2 - 733 (GCs= 2)
Line 374: 2014-09-29 19:13:00,458 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,734 (GCs= 2)
Line 376: 2014-09-29 19:13:01,153 [Runner thread] DEBUG Wintellect.OperationTimer2 - 665 (GCs= 2)
Line 378: 2014-09-29 19:13:01,805 [Runner thread] DEBUG Wintellect.OperationTimer2 - 627 (GCs= 2)
Line 380: 2014-09-29 19:13:02,406 [Runner thread] DEBUG Wintellect.OperationTimer2 - 559 (GCs= 2)
Line 382: 2014-09-29 19:13:02,834 [Runner thread] DEBUG Wintellect.OperationTimer2 - 396 (GCs= 2)
Line 384: 2014-09-29 19:13:03,476 [Runner thread] DEBUG Wintellect.OperationTimer2 - 615 (GCs= 2)
Line 386: 2014-09-29 19:13:14,271 [Runner thread] DEBUG Wintellect.OperationTimer2 - 10,763 (GCs= 2)
Line 388: 2014-09-29 19:13:14,995 [Runner thread] DEBUG Wintellect.OperationTimer2 - 693 (GCs= 2)
Line 390: 2014-09-29 19:13:20,586 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,559 (GCs= 3)
Line 392: 2014-09-29 19:13:21,353 [Runner thread] DEBUG Wintellect.OperationTimer2 - 731 (GCs= 2)
Line 394: 2014-09-29 19:13:21,933 [Runner thread] DEBUG Wintellect.OperationTimer2 - 550 (GCs= 2)
Line 396: 2014-09-29 19:13:22,508 [Runner thread] DEBUG Wintellect.OperationTimer2 - 542 (GCs= 2)
Line 398: 2014-09-29 19:13:23,228 [Runner thread] DEBUG Wintellect.OperationTimer2 - 689 (GCs= 2)
Line 400: 2014-09-29 19:13:23,739 [Runner thread] DEBUG Wintellect.OperationTimer2 - 477 (GCs= 2)
Line 402: 2014-09-29 19:13:29,418 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,649 (GCs= 3)
Line 404: 2014-09-29 19:13:30,048 [Runner thread] DEBUG Wintellect.OperationTimer2 - 590 (GCs= 2)
Line 406: 2014-09-29 19:13:30,639 [Runner thread] DEBUG Wintellect.OperationTimer2 - 557 (GCs= 2)
Line 408: 2014-09-29 19:13:31,193 [Runner thread] DEBUG Wintellect.OperationTimer2 - 524 (GCs= 2)
Line 410: 2014-09-29 19:13:31,825 [Runner thread] DEBUG Wintellect.OperationTimer2 - 599 (GCs= 2)
Line 412: 2014-09-29 19:13:32,429 [Runner thread] DEBUG Wintellect.OperationTimer2 - 573 (GCs= 2)
Line 414: 2014-09-29 19:13:48,187 [Runner thread] DEBUG Wintellect.OperationTimer2 - 15,732 (GCs= 3)
Line 416: 2014-09-29 19:13:48,806 [Runner thread] DEBUG Wintellect.OperationTimer2 - 578 (GCs= 2)
Line 418: 2014-09-29 19:13:49,433 [Runner thread] DEBUG Wintellect.OperationTimer2 - 591 (GCs= 2)
Line 420: 2014-09-29 19:13:50,060 [Runner thread] DEBUG Wintellect.OperationTimer2 - 594 (GCs= 2)
Line 422: 2014-09-29 19:13:50,912 [Runner thread] DEBUG Wintellect.OperationTimer2 - 807 (GCs= 2)
Line 424: 2014-09-29 19:13:51,553 [Runner thread] DEBUG Wintellect.OperationTimer2 - 609 (GCs= 2)
Line 426: 2014-09-29 19:13:52,447 [Runner thread] DEBUG Wintellect.OperationTimer2 - 868 (GCs= 2)
Line 428: 2014-09-29 19:13:53,022 [Runner thread] DEBUG Wintellect.OperationTimer2 - 541 (GCs= 2)
Line 430: 2014-09-29 19:13:53,711 [Runner thread] DEBUG Wintellect.OperationTimer2 - 649 (GCs= 2)
Line 432: 2014-09-29 19:13:59,399 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,655 (GCs= 3)
Line 434: 2014-09-29 19:14:00,317 [Runner thread] DEBUG Wintellect.OperationTimer2 - 882 (GCs= 2)
Line 436: 2014-09-29 19:14:00,949 [Runner thread] DEBUG Wintellect.OperationTimer2 - 600 (GCs= 2)
Line 438: 2014-09-29 19:14:01,647 [Runner thread] DEBUG Wintellect.OperationTimer2 - 669 (GCs= 2)
Line 440: 2014-09-29 19:14:02,398 [Runner thread] DEBUG Wintellect.OperationTimer2 - 712 (GCs= 2)
Line 442: 2014-09-29 19:14:03,046 [Runner thread] DEBUG Wintellect.OperationTimer2 - 616 (GCs= 2)
Line 444: 2014-09-29 19:14:03,539 [Runner thread] DEBUG Wintellect.OperationTimer2 - 462 (GCs= 2)
Line 446: 2014-09-29 19:14:03,965 [Runner thread] DEBUG Wintellect.OperationTimer2 - 395 (GCs= 2)
Line 448: 2014-09-29 19:14:04,393 [Runner thread] DEBUG Wintellect.OperationTimer2 - 395 (GCs= 2)
Line 450: 2014-09-29 19:14:04,980 [Runner thread] DEBUG Wintellect.OperationTimer2 - 562 (GCs= 2)
Line 452: 2014-09-29 19:14:10,189 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,168 (GCs= 2)
Line 454: 2014-09-29 19:14:10,368 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 456: 2014-09-29 19:14:10,555 [Runner thread] DEBUG Wintellect.OperationTimer2 - 162 (GCs= 2)
Line 458: 2014-09-29 19:14:10,714 [Runner thread] DEBUG Wintellect.OperationTimer2 - 122 (GCs= 2)
Line 460: 2014-09-29 19:14:10,920 [Runner thread] DEBUG Wintellect.OperationTimer2 - 175 (GCs= 2)
Line 462: 2014-09-29 19:14:11,088 [Runner thread] DEBUG Wintellect.OperationTimer2 - 142 (GCs= 2)
Line 464: 2014-09-29 19:14:11,240 [Runner thread] DEBUG Wintellect.OperationTimer2 - 120 (GCs= 2)
Line 466: 2014-09-29 19:14:11,443 [Runner thread] DEBUG Wintellect.OperationTimer2 - 174 (GCs= 2)
Line 468: 2014-09-29 19:14:11,620 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 470: 2014-09-29 19:14:11,805 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 472: 2014-09-29 19:14:12,008 [Runner thread] DEBUG Wintellect.OperationTimer2 - 174 (GCs= 2)
Line 474: 2014-09-29 19:14:12,190 [Runner thread] DEBUG Wintellect.OperationTimer2 - 158 (GCs= 2)
Line 476: 2014-09-29 19:14:12,353 [Runner thread] DEBUG Wintellect.OperationTimer2 - 132 (GCs= 2)
Line 478: 2014-09-29 19:14:12,530 [Runner thread] DEBUG Wintellect.OperationTimer2 - 149 (GCs= 2)
Line 480: 2014-09-29 19:14:12,724 [Runner thread] DEBUG Wintellect.OperationTimer2 - 171 (GCs= 2)
Line 482: 2014-09-29 19:14:12,883 [Runner thread] DEBUG Wintellect.OperationTimer2 - 126 (GCs= 2)
Line 484: 2014-09-29 19:14:13,053 [Runner thread] DEBUG Wintellect.OperationTimer2 - 136 (GCs= 2)
Line 486: 2014-09-29 19:14:13,221 [Runner thread] DEBUG Wintellect.OperationTimer2 - 145 (GCs= 2)
Line 488: 2014-09-29 19:14:13,349 [Runner thread] DEBUG Wintellect.OperationTimer2 - 96 (GCs= 2)

Note that all of those requests were against the node that was not removed from the cluster. They were also the same view query.




[NCBC-661] When the only operation are view requests, updated configs are not sent to client Created: 28/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When the client has boot straps using "Optimized configuration management" and the only operation being performed, the client never updates it's server configuration, because it doesn't have any means of knowing that it has changed.

 Comments   
Comment by Jeff Morris [ 29/Sep/14 ]
Two Scenarios:
1. Node is added while load is view specific
2. Node is removed while load is view specific

Note that when a node is removed, the server first returns the following message:

Error: noproc. Reason: {gen_server,call,[request_throttler,{note_request,<0.8540.71>,capi},infinity]}.

For several requests, followed by:

{"error":"not_found","reason":"missing"}

For scenario #1, there is no way for the client to be aware of the added node unless it polls or receives a config update asynchronously. For #2, we could use the response message, but the "noproc" message is only returned for a short time and the "not_found" message is the same message returned when the bucket doesn't exist - if the view or design doc doesn't, it's noted in the error message. For example:

{"error":"not_found","reason":"Error opening view `brewery_beers2`, from set `beer-sample`, design document `_design/beer`: {not_found,\n missing_named_view}"}






[NCBC-655] .NET Client seeing many pauses during Rebalance Created: 23/Sep/14  Updated: 29/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 1.3.9
Fix Version/s: backlog-1.0

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I have been benchmarking the Java client vs .NET client. During a rebalance, I see frequent pauses within the .NET client that greatly impacts overall throughput. These pauses are not seen in the Java client with equivalent code (gets). I've captured a 30 second video as well as a 30 second network capture, attached. It is connecting to a 3 node 2.5.1 cluster where I add or remove a node, and rebalance.

ASK: Why is the .NET client pausing for 10-15 seconds each minute during a rebalance?

http://screencast.com/t/kNyXkauaEPK3

Here is the relevant C# code, performing a GetWithCas with random keys in a loop:

private static void Main(string[] args)
        {
         
            var config = new CouchbaseClientConfiguration();

            config.Bucket = "TestBucket";
            config.Username = "";
            config.Password = "";

            config.SocketPool.MaxPoolSize = 20;
            config.SocketPool.MinPoolSize = 20;
            config.RetryCount = 10;
            config.HeartbeatMonitor.Interval = 30000;
            config.SocketPool.ReceiveTimeout = TimeSpan.FromMilliseconds(2500);
            config.RetryTimeout = TimeSpan.FromMilliseconds(1000);

            config.Urls.Add(new Uri("http://10.4.2.57:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.107:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.108:8091/pools"));

            var Client = new CouchbaseClient(config);

            // create threads to read random documents:

            ParallelOptions parallelOptions = new ParallelOptions();
            parallelOptions.MaxDegreeOfParallelism = 1;
            Parallel.For((long)0, 1, parallelOptions, i =>
            {
                for (int j = 100071; j < 1000000; j++)
                {
                    string key = "Couch_" + j.ToString();
                    var casVal = Client.GetWithCas(key);
                    if (casVal.Result != null)
                    {
                        Console.WriteLine("j=" + j.ToString());
                    }
                }
            });

            Client.Dispose();
        }
    }


FYI, here is the corresponding Java code that does not paused during a rebalance:

public static void main(String[] args) throws Exception {
        new App().run();
    }
    
    private void run() throws InterruptedException, ExecutionException {
        int numThreads = 1;
        
        ExecutorService pool = Executors.newFixedThreadPool(numThreads);
        List<Future> fs = new ArrayList<Future>(numThreads);
        for (int i=0; i<numThreads; i++) {
            fs.add(pool.submit(new Reader(i+1, 5000000)));
        }
        for (Future f : fs) {
            f.get();
        }
    }

    private class Reader implements Runnable {
        
        private int id;
        private long runningTime;
        
        public Reader(int id, long runningTime) {
            this.id = id;
            this.runningTime = runningTime;
        }
        
        @Override
        public void run() {
            long n=100071;
            long start, end=0;
            long initTime = System.currentTimeMillis();
            List<URI> uris = null;
            try {
                uris = Arrays.asList(new URI("http://10.4.2.57:8091/pools"));
            } catch (URISyntaxException e) {
                
                e.printStackTrace();
            }
            String bucketName = "TestBucket";
            String result = "";
            String password = "";
            CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
            builder.setReadBufferSize(20000);
            builder.setOpTimeout(2500);
            builder.setFailureMode(FailureMode.Redistribute);
            builder. setOpQueueMaxBlockTime(1000);
            CouchbaseConnectionFactory connectionFactory = null;
            CouchbaseClient client = null;
            try {
                connectionFactory =
                builder.buildCouchbaseConnection(uris, bucketName, "", "");
                client = new com.couchbase.client.CouchbaseClient(connectionFactory);
            } catch (IOException e) {
                e.printStackTrace();
            }
            
            while (true) {
                String key = "Couch_" + n;
                start = System.currentTimeMillis();
                
                if (client.gets(key) !=null) {
                    System.out.println("N=" + n);
                }
                
                n++;
                if (runningTime>0 && (end-initTime)>runningTime) {
                    break;
                }
            }
        }
    }


 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
Jeff -

Thanks for doing this :) Not sure why the client pauses - I have some ideas, though; I'll have to take a deeper look.

-Jeff




[NCBC-660] Exception: SafeHandle has been closed in MemcachedBucket.MultiGet method. Created: 28/Sep/14  Updated: 28/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at System.Security.Cryptography.Utils.HashData(SafeHashHandle hHash, Byte[] data, Int32 cbData, Int32 ibStart, Int32 cbSize)
   at System.Security.Cryptography.MD5CryptoServiceProvider.HashCore(Byte[] rgb, Int32 ibStart, Int32 cbSize)
   at System.Security.Cryptography.HashAlgorithm.ComputeHash(Byte[] buffer)
   at Couchbase.Core.Buckets.KetamaKeyMapper.GetHash(String key) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\KetamaKeyMapper.cs:line 80
   at Couchbase.Core.Buckets.KetamaKeyMapper.MapKey(String key) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\KetamaKeyMapper.cs:line 38
   at Couchbase.Core.Buckets.MemcachedBucket.GetServer(String key) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\MemcachedBucket.cs:line 100
   at Couchbase.Core.Buckets.MemcachedBucket.SendWithRetry[T](IOperation`1 operation) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\MemcachedBucket.cs:line 110
   at Couchbase.Core.Buckets.MemcachedBucket.Get[T](String key) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\MemcachedBucket.cs:line 595
   at Couchbase.Core.Buckets.MemcachedBucket.<>c__DisplayClass23`1.<Get>b__22(Tuple`2 range, ParallelLoopState loopstate) in c:\Users\jmorris\repos\CouchbaseNetClient-Build-2.0\couchbase-net-client\Src\Couchbase\Core\Buckets\MemcachedBucket.cs:line 614
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32`2.<PartitionerForEachWorker>b__30()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)




[NCBC-648] Forcing JsonSerialization on non-nullable ValueTypes fails Created: 18/Sep/14  Updated: 28/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If you try do to a Set on a non-nullable ValueType, NewtonSoft will throw an exception with the following error message returned in the IResult object: "No JSON content found and type 'System.Int32' is not nullable. Path '', line 0, position 0."

The work around is to use a nullable valuetype instead, for example use bucket.Insert<int?>(..) instead of bucket.Insert<int>(..).

 Comments   
Comment by Jeff Morris [ 28/Sep/14 ]
This might be resolvable through proper documentation.




[NCBC-659] KetamaKeyMapper needs to implement IDisposable Created: 28/Sep/14  Updated: 28/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This class maintains a reference to HashAlgorithm which has a Dipose method which is not called.




[NCBC-640] Couchbase.CouchbaseCluster is redundant. Created: 12/Sep/14  Updated: 28/Sep/14  Resolved: 28/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0.0

Type: Task Priority: Major
Reporter: Brett Lawson Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The CouchbaseCluster class duplicates the namespace. Should this class not be named simply 'Cluster'?

 Comments   
Comment by Jeff Morris [ 28/Sep/14 ]
http://review.couchbase.org/#/c/41742/




[NCBC-658] Fix bug where "/pools" is not added to bootstrap urls when configuration is app.config Created: 28/Sep/14  Updated: 28/Sep/14  Resolved: 28/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[NCBC-611] CouchbaseAspNet 1.2.1 Nuget package was released built in debug mode Created: 28/Aug/14  Updated: 26/Sep/14  Resolved: 26/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: None
Fix Version/s: backlog-1.0

Type: Bug Priority: Major
Reporter: Stephen Phillips Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
v1.2.1 of "Couchbase ASP.NET SessionState and OutputCache Providers" on nuget the dll included in the package was built in debug mode and was raised to us when we sent a dump file to microsoft for anaylsis. This has been suggested as a potential performance issue. An updated release build would be appreciated.

 Comments   
Comment by Stephen Phillips [ 28/Aug/14 ]
Apologies if this is the wrong project I did not know where else to create the issue.
Comment by Jeff Morris [ 28/Aug/14 ]
Thanks Stephen, I'll look into it. It's fine to create issues for that project here :)
Comment by Stephen Phillips [ 26/Sep/14 ]
Hi, any update on this?
Comment by Jeff Morris [ 26/Sep/14 ]
Release notes
--------------------
- Updated to CouchbaseNetClient 1.3.1*
- Packaged binaries are now release binaries

*It's recommended that you update to a later version of the client using Nuget




[NCBC-649] infinite loop 'Trying again' with 'No connections currently available' for ReAdd2 test when nodes returned to cluster Created: 19/Sep/14  Updated: 26/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-649_1.png     PNG File NCBC-649_2.png     PNG File NCBC-649_3.png     PNG File NCBC-649_4.png     File NCBC-649.7z     File NCBC-649_gerrit.7z     PNG File NCBC-649.png    

 Description   
run with commit https://github.com/couchbase/couchbase-net-client/commit/24714876f76207a5fc7d72e88bd1f9c57909fcdb

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/102/console

Running ReAdd2-HYBRID: 2014-09-18 23:50:57
    workload=HybridWorkloadGroup
    failover/next_action=FO_READD_REBALANCE
    failover/count=2
    failover/next_delay=45
    testcase=FailoverScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\ReAdd2-HYBRID\09-18-14\024657\8c847f52905fd0bf669803f16c2dc716

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
        --workload HybridWorkloadGroup --failover-next_action
 \
        FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
 \
        --testcase FailoverScenario -C share\rexec --rexec_path
 \
        C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \
        --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \
        --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \
        --cluster_ssh-username=root --cluster_ssh-password=couchbase


2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,614 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 1 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,614 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#878 on thread 42
2014-09-18 23:56:46,614 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 609 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 - rev878
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP4463REP4463REP4463REP4463REP4463 using server 10.3.121.134:11210
2014-09-18 23:56:46,615 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b | 10.3.121.134:11210 | [0, 2] - b20c68a7-f922-486e-af16-f880d88bfff6 - Disposed: False
2014-09-18 23:56:46,615 [42] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP4463REP4463REP4463REP4463REP4463 using 609 on 10.3.121.134:11210
2014-09-18 23:56:46,616 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b on 10.3.121.134:11210 - b20c68a7-f922-486e-af16-f880d88bfff6
2014-09-18 23:56:46,619 [42] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1736
2014-09-18 23:56:46,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:46,880 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 2 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 failed after 3 retries. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:49,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:51,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:54,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:56,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:59,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:01,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:04,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


..................


2014-09-19 01:09:51,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:54,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:56,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:59,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:10:01,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


test stuck, not all items loaded
run was via jenkins, so I'm not able to get stacktraces



 Comments   
Comment by Jeff Morris [ 19/Sep/14 ]
Hmm, looks like connection pool exhausted, but none of the in-progress operations are timing out? Strange.
Comment by Andrei Baranouski [ 19/Sep/14 ]
right, something like that
Comment by Andrei Baranouski [ 20/Sep/14 ]
catched it in manual run


  [In a sleep, wait, or join]
  mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) + 0x22 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) + 0x28 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout) + 0xd bytes
> Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 123 + 0x28 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
 
        .................................
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Couchbase.IO.IConnectionPool.Acquire() Line 178 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.Strategies.DefaultIOStrategy.Execute<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 40 + 0xc bytes C#
  Couchbase.dll!Couchbase.Core.Server.Send<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 88 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 103 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.Upsert<object>(string key, object value) Line 332 + 0x31 bytes C#
  Sdkd.dll!Sdkd.Commands.SetCommand.ExecuteSingle(string key, string value) Line 78 + 0x1e bytes C#
  Sdkd.dll!Sdkd.Commands.KVCommand.Run() Line 24 + 0x36 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.Buckets.CouchbaseBucket cli) Line 266 + 0x9 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 146 + 0xf bytes C#
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x6f bytes
  mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0xa7 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x16 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x41 bytes
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes
  [Native to Managed Transition]


please note, this run was with some commits from gerrit

$ git log
commit 8afa0ace0ac366e53dcc07a62c23ffd6604f6015
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

    Change-Id: I41bd3d93b31b98a19b8fb23d432713f5034debd7

commit 602181eed21614451ccaf0f848337f9c382ae578
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 00:10:53 2014 -0700

    NCBC-590: Refactor logging so that ODE are not raised when it references a disposed object

    Change-Id: I5d81dba00d64e4caf34d165491624e2c02a9949b

commit f9d4efb511c93a64b2a844e41849177da1f4442a
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:53:12 2014 -0700

    Refactor unit tests to reflect changes in exceptions when bootstrap fails

    Change-Id: I387aa3c8a0d8954102265e27dbbe4f6d84c4d104

commit 2a054aceab491ed338f1fc435d0a48eb4d31c43c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:13:02 2014 -0700

    NCBC-624: Refactor View API

    From Michael's review:

    "One thing that I was confused about was the View query DSL. The required
    argument is development, but the view and design document name could be
    omitted by the DSL user since they are elements like limit and so forth. I
    think it would be better to turn this around. For example in Java there is
    ViewQuery.from('designDoc', 'view') and then the DSL starts.
    So you can never miss design doc and view name by accident. And I think
    you can go with a development = false default, since that's what people
    use most of the time. I think the main question is, if I miss adding
    design doc or view name by accident, does it blow up at runtime or at
    compile time?"

    Change-Id: I4fc9d716f674df882156cdd1763b8d43204e7d90

commit 24714876f76207a5fc7d72e88bd1f9c57909fcdb
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 19:06:31 2014 -0700

    NCBC-606: Catch and log ServerUnavailableException during failover

    Change-Id: Ie0f74d2bf69cd43ea399762fc39e7ca7f305c11c
    Reviewed-on: http://review.couchbase.org/41384
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit a2cd618f125fc330db742bcb6879c85741e1b3e0
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:43:48 2014 -0700

    NCBC-638: If Server is null when sending an operation then retry operation

    Change-Id: Idfee0ab2852c51cb9c7de1912374d64ee1633be7
    Reviewed-on: http://review.couchbase.org/41374
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 2ca77982d595f15cfa7fe5aa0b97902c94b5f120
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:41:13 2014 -0700

    Update unit tests to use camel cased property values when applicable

    Change-Id: I80ae31e2bf58809574dba965063b21ffe48dc5ec
    Reviewed-on: http://review.couchbase.org/41373
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 33fc515c0248fbeb56fa7123cea954872e4c9b11
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:35:10 2014 -0700

    NCBC-637: When CCCP fails ensure NRE is not thrown

    Change-Id: I5ef0dbef8298675ceb92e9eb91c49303269133a2
    Reviewed-on: http://review.couchbase.org/41371
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 1e9fff367fa3eefd19069fdfa1dfaae14b4b2a42
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 23:36:43 2014 -0700

    NCBC-586: Make AddRef and Release on IBucket not public

    This commit adds an internal interface IRefCountable with AddRef and
    Release members, so that the two methods are not publically callable on an
    IBucket implementation.

    Change-Id: I11f628094aa1f65ff4780f42e00bbfb088e78898
    Reviewed-on: http://review.couchbase.org/41354
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 983b94b8acbce8eee11314542d8dcdfbb3c03e0d
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 22:34:37 2014 -0700

    NCBC-636: Make outgoing ContractResolver default to camel casing

    This commit makes the incoming and outgoing JSON IContractResolver's type
    configurable from the ClientConfiguration object. The default settings are
    to use a CamelCasedNameContractResolver for both serialization and
    deserilization.

    Change-Id: I3a82f95d5f4608aa8d05e4141a1a089bc300d4e7
    Reviewed-on: http://review.couchbase.org/41353
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 77b78aaa1550c2ea9e0dfa4085fad2cd24814721
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 18:06:55 2014 -0700

    NCBC-635: Provide better failure support for N1QL Query API

    Change-Id: I16d9a44ba653f8846b22e7c679681ab262e1e6bc
    Reviewed-on: http://review.couchbase.org/41351
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 424a10647f5d6dd21e603fbfe8bb98de305f9491
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:43:24 2014 -0700

    NCBC-634: Ensure that CancellationTokenSource is Disposed

    This fix ensures that the CTS will be explicitly disposed so that the
    object does not live through to finalization.

    Change-Id: I39545b97a3c934f2c0a93587fa7147b32d2bd043
    Reviewed-on: http://review.couchbase.org/41335
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8bd607e8cb545ace5102edf1accb5f862b5caf74
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:12:17 2014 -0700

    NCBC-632: Make JsonSerializerSettings configurable

    Change-Id: I21c5620a2192217faa68a62cf440a2a8f3dc8893
    Reviewed-on: http://review.couchbase.org/41331
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit bebae67219460e2e3a69ad2d56c9f98afc824190
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 14:26:25 2014 -0700

    NCBC-626: HttpRequestException in queries after rebalance out

    Change-Id: I0eed00d956f231c3826a54da6a983586618d126d
    Reviewed-on: http://review.couchbase.org/41327
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8db6085e9aecb23cd6defb672f7ec0548536d9de
Author: jeffrymorris <jeffrymorris@gmail.com>

http://www.couchbase.com/issues/secure/attachment/21870/NCBC-649_gerrit.7z


please see screenshots
Comment by Andrei Baranouski [ 26/Sep/14 ]
still see the issue with https://github.com/couchbase/couchbase-net-client/commit/72b06be3aa7b6ae8ed478f006fd7ebe89f3cae89
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/106/console 2.5.0-1059




[NCBC-626] HttpRequestException in queries after rebalance out Created: 07/Sep/14  Updated: 25/Sep/14  Resolved: 25/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0.0

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1208

Attachments: HTML File 7c866d0c3984e2387129b6f66f32f19f-CB.html     HTML File 7c866d0c3984e2387129b6f66f32f19f-HT.html     Zip Archive NCBC-626.log.zip     Zip Archive NCBC-626_new.log.zip     File NCBC-626_with_gerrit.7z     Zip Archive NCBC-626_without_ssl.log.zip     Text File test_logs_new.txt     Text File test.txt    

 Description   
c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,12 INFO] (BRun run:441) Initializing history database

============================================================
Running Rb2Out-HYBRID: 2014-09-06 20:26:43
    rebalance/count=2
    workload=HybridWorkloadGroup
    rebalance/mode=out
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\Rb2Out-HYBRID\09-06-14\062803\7c866d0c3984e2387129b6f66f32f19f

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-count 2 --workload HybridWorkloadGroup
 \ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_useSSL=True --cluster_ssh-username=root
        --cluster_ssh-password=couchbase
    ---------------------------------------->
....


[4250,29 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[4251,51 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[4252,16 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4252,16 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4252,16 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268

[4252,72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[4253,96 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 84%
[4255,18 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 84%
[4256,39 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[4257,60 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[4258,82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[4260,04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 86%
[4261,25 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 86%
[4262,49 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 86%
[4263,71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[4264,94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[4266,15 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[4267,37 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 88%
[4268,58 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 88%
[4269,81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 88%
[4271,04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[4272,26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[4273,48 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[4274,70 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[4275,95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[4277,17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[4278,41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[4279,63 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[4280,85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[4282,09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[4283,31 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[4284,52 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[4285,73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[4286,95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 93%
[4288,17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 94%
[4289,39 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 94%
[4290,62 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 96%
[4292,55 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 98%
[4292,89 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4292,89 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4293,81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 99%
[4295,06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4296,28 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4297,50 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4298,72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4300,01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4301,23 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4302,45 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4303,66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4304,88 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[4306,17 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[4306,17 INFO] (Scenario run:97) CHANGE phase done
[4306,17 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[4307,52 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4307,52 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4307,57 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4307,57 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4308,06 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4308,06 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4308,43 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4308,43 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4309,46 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4309,46 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268


sdkd gets these exceptions on https://github.com/couchbase/sdkd-net/blob/client-2.0/src/Sdkd/Views/ViewCommand.cs#L268
it means that 'IViewResult<dynamic> view = GetViewObject(Client, Req, MyOptions);' returns null

GetViewObject has been implemented this way:
https://github.com/couchbase/sdkd-net/blob/client-2.0/src/Sdkd/Views/ViewCommand.cs#L164




from client logs I watch a lot of errors:

2014-09-06 21:39:58,200 [35] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 451 for key SimpleKeyREP3579REP3579REP3579REP3579REP3579 - rev4310
2014-09-06 21:39:58,200 [35] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP3579REP3579REP3579REP3579REP3579 using server 10.3.3.206:11207
2014-09-06 21:39:58,200 [35] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 55527dd8-1b82-4208-9ff1-86743b0ebc7e | 10.3.3.206:11207 | [0, 2] - b5c60b0f-f3aa-406e-a97d-7eee28bdc5ae - Disposed: False
2014-09-06 21:39:58,200 [18] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: he remote certificate is invalid according to the validation procedure.
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.PooledStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---

 Comments   
Comment by Andrei Baranouski [ 07/Sep/14 ]
observe the same situation without ssl, however client logs do not have any exceptions. only sвkd issues the same exceptions:
Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4308,43 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
Comment by Jeff Morris [ 09/Sep/14 ]
The NullReferenceException is because the view request failed, thus IViewResult.Rows is null, but your trying to iterate over it in your foreach list which throws a NRE. The reason why the view request is failing is because they are being attempted on nodes that are either out of the cluster or in a funky state.

A couple of things to be improved upon:
a) The Message field correctly gives the reason (good)
b) The StatusCode field is defaulting to found (200) - this is invalid (bad)
c) The Exception field is null (bad)
b) Rows can default to an empty list, which is probably better than null (design choice)
e) There is no retry logic for views like there is for memcached operations - this is a missing feature.(bad)

I'll post a changeset asap.

-Jeff



Comment by Jeff Morris [ 10/Sep/14 ]
http://review.couchbase.org/#/c/41327/
Comment by Andrei Baranouski [ 17/Sep/14 ]
Hi Jeff,

Hi Jeff,

seems like the case "b) Rows can default to an empty list, which is probably better than null (design choice) " still doesn't work with commit

commit 7e1a74cded3201aa6cfa702597cd1a97a94e6d9e
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

    Change-Id: I41bd3d93b31b98a19b8fb23d432713f5034debd7

in our functional python tests we we always make sure that the query returns something without any errors (maybe it will be an empty list , ie rows =0), but as I understand that return zero is unacceptable for us



4611,07 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[4611,07 INFO] (Scenario run:97) CHANGE phase done
[4611,07 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[4612,19 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4612,19 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4612,20 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268

[4612,34 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4612,34 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[4613,39 INFO] (SDKD log:137) Exception: System.NullReferenceException: object reference not set to an instance of an object.
[4613,39 INFO] (SDKD log:137) at Sdkd.Views.ViewCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268


sdkd gets these exceptions on https://github.com/couchbase/sdkd-net/blob/client-2.0/src/Sdkd/Views/ViewCommand.cs#L268
it means that 'IViewResult<dynamic> view = GetViewObject(Client, Req, MyOptions);' returns null

please see new logs




Comment by Jeff Morris [ 17/Sep/14 ]
Let me check the case of View that returns no rows. I'll post the results asap.
Comment by Jeff Morris [ 17/Sep/14 ]
Andrei -

This should be fixed with the commit for NCBC-626. In the comment above you referenced NCBC-639 which has yet been through code review. Once it's merged with master it should be resolved.
Comment by Jeff Morris [ 17/Sep/14 ]
This isn't merged with Master yet.
Comment by Andrei Baranouski [ 17/Sep/14 ]
hm, my current results were based on latest master branch + commits in gerrit till
commit 7e1a74cded3201aa6cfa702597cd1a97a94e6d9e
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

in any case, I'll check on the master since now you've added a lot of commits
Comment by Andrei Baranouski [ 19/Sep/14 ]
Hi Jeff, today I run the tests with the latest commits from gerrit

$ git log
commit 8afa0ace0ac366e53dcc07a62c23ffd6604f6015
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

    Change-Id: I41bd3d93b31b98a19b8fb23d432713f5034debd7

commit 602181eed21614451ccaf0f848337f9c382ae578
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 00:10:53 2014 -0700

    NCBC-590: Refactor logging so that ODE are not raised when it references a disposed object

    Change-Id: I5d81dba00d64e4caf34d165491624e2c02a9949b

commit f9d4efb511c93a64b2a844e41849177da1f4442a
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:53:12 2014 -0700

    Refactor unit tests to reflect changes in exceptions when bootstrap fails

    Change-Id: I387aa3c8a0d8954102265e27dbbe4f6d84c4d104

commit 2a054aceab491ed338f1fc435d0a48eb4d31c43c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:13:02 2014 -0700

    NCBC-624: Refactor View API

    From Michael's review:

    "One thing that I was confused about was the View query DSL. The required
    argument is development, but the view and design document name could be
    omitted by the DSL user since they are elements like limit and so forth. I
    think it would be better to turn this around. For example in Java there is
    ViewQuery.from('designDoc', 'view') and then the DSL starts.
    So you can never miss design doc and view name by accident. And I think
    you can go with a development = false default, since that's what people
    use most of the time. I think the main question is, if I miss adding
    design doc or view name by accident, does it blow up at runtime or at
    compile time?"

    Change-Id: I4fc9d716f674df882156cdd1763b8d43204e7d90

commit 24714876f76207a5fc7d72e88bd1f9c57909fcdb
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 19:06:31 2014 -0700

    NCBC-606: Catch and log ServerUnavailableException during failover

    Change-Id: Ie0f74d2bf69cd43ea399762fc39e7ca7f305c11c
    Reviewed-on: http://review.couchbase.org/41384
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit a2cd618f125fc330db742bcb6879c85741e1b3e0
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:43:48 2014 -0700

    NCBC-638: If Server is null when sending an operation then retry operation

    Change-Id: Idfee0ab2852c51cb9c7de1912374d64ee1633be7
    Reviewed-on: http://review.couchbase.org/41374
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 2ca77982d595f15cfa7fe5aa0b97902c94b5f120
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:41:13 2014 -0700

    Update unit tests to use camel cased property values when applicable

    Change-Id: I80ae31e2bf58809574dba965063b21ffe48dc5ec
    Reviewed-on: http://review.couchbase.org/41373
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 33fc515c0248fbeb56fa7123cea954872e4c9b11
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:35:10 2014 -0700

    NCBC-637: When CCCP fails ensure NRE is not thrown

    Change-Id: I5ef0dbef8298675ceb92e9eb91c49303269133a2
    Reviewed-on: http://review.couchbase.org/41371
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 1e9fff367fa3eefd19069fdfa1dfaae14b4b2a42
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 23:36:43 2014 -0700

    NCBC-586: Make AddRef and Release on IBucket not public

    This commit adds an internal interface IRefCountable with AddRef and
    Release members, so that the two methods are not publically callable on an
    IBucket implementation.

    Change-Id: I11f628094aa1f65ff4780f42e00bbfb088e78898
    Reviewed-on: http://review.couchbase.org/41354
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 983b94b8acbce8eee11314542d8dcdfbb3c03e0d
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 22:34:37 2014 -0700

    NCBC-636: Make outgoing ContractResolver default to camel casing

    This commit makes the incoming and outgoing JSON IContractResolver's type
    configurable from the ClientConfiguration object. The default settings are
    to use a CamelCasedNameContractResolver for both serialization and
    deserilization.

    Change-Id: I3a82f95d5f4608aa8d05e4141a1a089bc300d4e7
    Reviewed-on: http://review.couchbase.org/41353
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 77b78aaa1550c2ea9e0dfa4085fad2cd24814721
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 18:06:55 2014 -0700

    NCBC-635: Provide better failure support for N1QL Query API

    Change-Id: I16d9a44ba653f8846b22e7c679681ab262e1e6bc
    Reviewed-on: http://review.couchbase.org/41351
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 424a10647f5d6dd21e603fbfe8bb98de305f9491
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:43:24 2014 -0700

    NCBC-634: Ensure that CancellationTokenSource is Disposed

    This fix ensures that the CTS will be explicitly disposed so that the
    object does not live through to finalization.

    Change-Id: I39545b97a3c934f2c0a93587fa7147b32d2bd043
    Reviewed-on: http://review.couchbase.org/41335
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8bd607e8cb545ace5102edf1accb5f862b5caf74
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:12:17 2014 -0700

    NCBC-632: Make JsonSerializerSettings configurable

    Change-Id: I21c5620a2192217faa68a62cf440a2a8f3dc8893
    Reviewed-on: http://review.couchbase.org/41331
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit bebae67219460e2e3a69ad2d56c9f98afc824190
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 14:26:25 2014 -0700

    NCBC-626: HttpRequestException in queries after rebalance out

    Change-Id: I0eed00d956f231c3826a54da6a983586618d126d
    Reviewed-on: http://review.couchbase.org/41327
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8db6085e9aecb23cd6defb672f7ec0548536d9de
Author: jeffrymorris <jeffrymorris@gmail.com>


so, my version includes the fix for NCBC-626 (master + gerrit commits)

and I still see 'System.NullReferenceException: Object reference not set to an instance of an object' in Rb2Out test( after rebalance completed)
 http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/103/console

strange thing is that if run without a commit from gerrit, this problem is not present http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/97/consoleFull

{only one exception has been generated there (run Sep 17, 2014 1:22:20 AM with master)

[307.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 98%
[308.90 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 98%
[309.33 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[309.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[310.92 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[311.94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%

}


https://www.couchbase.com/issues/secure/attachment/21866/NCBC-626_with_gerrit.7z
Comment by Jeff Morris [ 25/Sep/14 ]
http://review.couchbase.org/#/c/41633/

Wait until merged to test, please.




[NCBC-99] Provide Error Handling Consistent with Other SDKs Created: 24/Aug/12  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Task Priority: Major
Reporter: Anonymous Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Provide Error Handling Consistent with Other SDKs and underlying protocol.

 Comments   
Comment by Matt Ingenthron [ 24/Aug/12 ]
Karen: there are a number of specific bugs filed and work underway on this, planned for 1.1.7. Given the context, I don't know if they'll ever be perfectly consistent, but that's okay.

Was there something specific you were concerned with here?




[NCBC-625] View do not work with dynamic type or strings on Mono Created: 06/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: backlog-2.0

Type: Improvement Priority: Minor
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
From Michaels review:

"I think because of my own fault I was not able to get views to work because there it was complaining about the dynamic part which needs a dll that is not included, and when I did a .Query<string> Newtonsoft complained it could not convert it to a string. But don’t mind on that one, I’m sure it works good with VS of course."




[NCBC-493] Implement Replica Read Capability Created: 29/May/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: backlog-2.0

Type: Task Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by NCBC-84 Replica read command Resolved
Sub-Tasks:
Key
Summary
Type
Status
Assignee
NCBC-568 Update IVBucket to support more than ... Technical task Resolved Jeff Morris  




[NCBC-535] Complete Awaitable Socket IO Engine Created: 26/Jun/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: .future

Type: Story Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Link: Provide Async TAP methods for all operations - view, kv, and n1ql

 Description   
This is required so that TAP methods are async all the way through. This allows the users of the client to await on an operation.




[NCBC-653] MemcachedBucket does not have SendWithRetry method Created: 20/Sep/14  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Jeff Morris [ 24/Sep/14 ]
http://review.couchbase.org/#/c/41629/




[NCBC-103] Provide Consistent Order of Parameters Created: 24/Aug/12  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Improvement Priority: Major
Reporter: Anonymous Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Provide consistent, predictable order for method parameters as other SDKS.




[NCBC-440] Cluster object needs to cluster level Authentication Created: 28/Apr/14  Updated: 24/Sep/14  Resolved: 24/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: New Feature Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: 2.0
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
While buckets are authenticated, cluster level authentication also needs to be done.

 Comments   
Comment by Jeff Morris [ 24/Sep/14 ]
Resolved within the ClusterManager and BucketManager.




[NCBC-657] Provide documentation in readme.md for running unit tests Created: 24/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: docs, library
Affects Version/s: 2.0 Beta
Fix Version/s: backlog-2.0

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
From Andrei: "could you add in Readme pre-setup requirements for unit tests: 1) list of buckets 2) ip addresses of CB clusters that are used"




[NCBC-656] Couchbase .NET SDK 1.X and 2.X cannot be used within the same project Created: 23/Sep/14  Updated: 24/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta, 1.3.9
Fix Version/s: None

Type: Bug Priority: Trivial
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
**It is likely that this ticket will never be resolved, but I am creating it so that it exists as a known issue**

Visual Studio blocks adding the reference because of the naming conflict of the dll; there is no resolution for this except renaming the dll to Couchbase.2.0.dll. Additionally, assuming that would work, there would likely have to be quite a bit of namespace and Type aliasing required as well.






[NCBC-594] Provide support for Management API Created: 20/Aug/14  Updated: 23/Sep/14  Resolved: 23/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Story Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Sub-Tasks:
Key
Summary
Type
Status
Assignee
NCBC-595 Create and Implement ClusterManager c... Technical task Resolved Jeff Morris  
NCBC-596 Create BucketManager class Technical task Resolved Jeff Morris  

 Description   
As an admin
I want to administrate my cluster
So that I can automate tasks such as adding and removing buckets.


 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
http://review.couchbase.org/#/c/41539/
Comment by Jeff Morris [ 23/Sep/14 ]
Pushed to gerrit, but I still am not comfortable with a few aspects of the API. This biggest one being the return types are bools and I feel should be something richer so that the developer can adjust accordingly instead of spelunking through logs to find what went wrong. Other concerns:

a) unit tests need refactoring
b) possible refactoring of duplicate code
c) parameters and return types are a bit raw for the design doc bits
d) urls need ssl support




Provide support for Management API (NCBC-594)

[NCBC-596] Create BucketManager class Created: 20/Aug/14  Updated: 23/Sep/14  Resolved: 23/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Technical task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Methods required:
-------------------------------
insertDesignDocument
updateDesignDocument
getDesignDocument
removeDesignDocument
listDesignDocuments
flush

https://docs.google.com/document/d/1HgVEJetcIfeIqviKC9zdlv_7IEkWpstatzxeydkLF3A/edit#heading=h.t4f3x16dy0fh




 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
http://review.couchbase.org/#/c/41539/




Provide support for Management API (NCBC-594)

[NCBC-595] Create and Implement ClusterManager class Created: 20/Aug/14  Updated: 23/Sep/14  Resolved: 23/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Technical task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Methods required:
addNode
removeNode
rebalance
listBuckets
createBucket
removeBucket


https://docs.google.com/document/d/1HgVEJetcIfeIqviKC9zdlv_7IEkWpstatzxeydkLF3A/edit#heading=h.t4f3x16dy0fh




 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
http://review.couchbase.org/#/c/41539/




Generated at Tue Sep 30 07:42:42 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.