[NCBC-589] ThrowInvalidOperationException in Couchbase.Configuration.CouchbaseConfigContext.LoadConfig during rebalance Created: 16/Aug/14  Updated: 21/Aug/14  Resolved: 19/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-589.zip    
Issue Links:
Gantt: finish-start

 Description   
I took a few commits ahead from gerrit


commit 2c00a9d3a397f5c1eddc56238cf4a19e6e6618dd
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 11 18:41:08 2014 -0700

    Add config files to tests

    Change-Id: Ie093eac1da4ffc742796ffd4df8a9c63cc1c9c2f



c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0.32 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-16-14\056660\742db0c42511883a3b8960d57d363be0

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1.52 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1.93 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1.95 INFO] (RunContext run:124) Starting cluster and driver
[1.96 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1.96 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1.96 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2.30 INFO] (SDKD log:137) --- Logging Self-Test ---
[2.31 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[2.31 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[2.32 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[2.32 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[2.32 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2.48 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2.49 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2.78 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.17929"}}
[4.34 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[4.43 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[4.43 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.43 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[4.43 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.94 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[4.95 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[4.95 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[4.97 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[4.97 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[4.97 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[4.98 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[4.98 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[5.18 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@7b0bbd42
[5.19 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@10d4f27
[5.19 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@b22c9df
[5.20 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@74bfd10a
[5.20 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@6ae2d0b2
[17.28 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[24.00 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[36.81 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[43.42 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[43.46 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[44.47 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[44.47 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[44.48 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[44.49 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[54.57 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[54.62 INFO] (RunContext run:143) Driver and cluster initialized
[54.65 INFO] (RunContext call:167) Running scenario..
[54.66 INFO] (Scenario run:72) Starting RAMP phase
[54.66 INFO] (Workload setupDesign:63) Creating design test_design
[54.93 INFO] (Workload setupDesign:80) Design creation done
[54.93 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[54.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[54.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[54.96 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[54.97 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[54.97 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[55.00 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[55.01 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[55.42 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[55.42 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12
}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[55.43 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[55.43 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[55.43 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[55.43 INFO] (SDKD log:137) "DS": {
[55.43 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[55.44 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[55.44 INFO] (SDKD log:137) "Count": 1000,
[55.44 INFO] (SDKD log:137) "Continuous": false,
[55.44 INFO] (SDKD log:137) "VSize": 512,
[55.44 INFO] (SDKD log:137) "Repeat": "rep",
[55.44 INFO] (SDKD log:137) "KSize": 12
[55.45 INFO] (SDKD log:137) },
[55.45 INFO] (SDKD log:137) "Schema": {
[55.45 INFO] (SDKD log:137) "InflateContent": "meh",
[55.45 INFO] (SDKD log:137) "InflateLevel": 40
[55.45 INFO] (SDKD log:137) },
[55.46 INFO] (SDKD log:137) "Options": {
[55.46 INFO] (SDKD log:137) "DelayMax": 0,
[55.46 INFO] (SDKD log:137) "DelayMin": 0,
[55.46 INFO] (SDKD log:137) "IterWait": 1,
[55.46 INFO] (SDKD log:137) "TimeRes": 0
[55.46 INFO] (SDKD log:137) }
[55.47 INFO] (SDKD log:137) }
[56.75 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[56.76 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[56.76 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[56.76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[56.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[56.76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[56.76 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[56.77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[56.77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[56.77 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[56.77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[56.78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[56.79 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[56.79 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@102.5 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12
}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[56.79 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[56.80 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[56.80 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[56.80 INFO] (SDKD log:137) "DS": {
[56.81 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[56.81 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[56.81 INFO] (SDKD log:137) "Count": 1000,
[56.81 INFO] (SDKD log:137) "Continuous": false,
[56.81 INFO] (SDKD log:137) "VSize": 512,
[56.82 INFO] (SDKD log:137) "Repeat": "rep",
[56.82 INFO] (SDKD log:137) "KSize": 12
[56.82 INFO] (SDKD log:137) },
[56.82 INFO] (SDKD log:137) "Schema": {
[56.82 INFO] (SDKD log:137) "InflateContent": "meh",
[56.83 INFO] (SDKD log:137) "InflateLevel": 40
[56.83 INFO] (SDKD log:137) },
[56.83 INFO] (SDKD log:137) "Options": {
[56.83 INFO] (SDKD log:137) "DelayMax": 0,
[56.83 INFO] (SDKD log:137) "DelayMin": 0,
[56.84 INFO] (SDKD log:137) "IterWait": 1,
[56.84 INFO] (SDKD log:137) "TimeRes": 0
[56.84 INFO] (SDKD log:137) }
[56.84 INFO] (SDKD log:137) }
[58.30 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[58.31 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@102.5
[58.31 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[58.31 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[58.31 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[58.31 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[58.32 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[58.32 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[58.32 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.32 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[58.32 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[58.32 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.34 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[58.34 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@103.8 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCou
nt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[58.35 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[58.35 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.35 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[58.35 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.35 INFO] (SDKD log:137) "DS": {},
[58.36 INFO] (SDKD log:137) "ViewParameters": {},
[58.36 INFO] (SDKD log:137) "DesignName": "test_design",
[58.36 INFO] (SDKD log:137) "Options": {
[58.36 INFO] (SDKD log:137) "DelayMax": 10,
[58.36 INFO] (SDKD log:137) "DelayMin": 1,
[58.37 INFO] (SDKD log:137) "IterWait": 1,
[58.37 INFO] (SDKD log:137) "ViewQueryCount": -1,
[58.37 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[58.37 INFO] (SDKD log:137) "TimeRes": 1
[58.38 INFO] (SDKD log:137) },
[58.38 INFO] (SDKD log:137) "ViewName": "test_view"
[58.38 INFO] (SDKD log:137) }
[58.38 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.38 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[58.38 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.40 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.9
[58.40 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@104.10 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[58.41 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[58.41 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.41 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[58.41 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.41 INFO] (SDKD log:137) "DS": {},
[58.42 INFO] (SDKD log:137) "ViewParameters": {},
[58.42 INFO] (SDKD log:137) "DesignName": "test_design",
[58.42 INFO] (SDKD log:137) "Options": {
[58.42 INFO] (SDKD log:137) "DelayMax": 10,
[58.42 INFO] (SDKD log:137) "DelayMin": 1,
[58.42 INFO] (SDKD log:137) "IterWait": 1,
[58.43 INFO] (SDKD log:137) "ViewQueryCount": -1,
[58.43 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[58.43 INFO] (SDKD log:137) "TimeRes": 1
[58.43 INFO] (SDKD log:137) },
[58.43 INFO] (SDKD log:137) "ViewName": "test_view"
[58.44 INFO] (SDKD log:137) }
[58.44 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.44 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[58.44 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.57 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.11
[58.58 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@105.12 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[58.58 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[58.58 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.13 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[58.59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.59 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.59 INFO] (SDKD log:137) "DS": {},
[58.59 INFO] (SDKD log:137) "ViewParameters": {},
[58.59 INFO] (SDKD log:137) "DesignName": "test_design",
[58.60 INFO] (SDKD log:137) "Options": {
[58.60 INFO] (SDKD log:137) "DelayMax": 10,
[58.60 INFO] (SDKD log:137) "DelayMin": 1,
[58.60 INFO] (SDKD log:137) "IterWait": 1,
[58.60 INFO] (SDKD log:137) "ViewQueryCount": -1,
[58.60 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[58.60 INFO] (SDKD log:137) "TimeRes": 1
[58.61 INFO] (SDKD log:137) },
[58.61 INFO] (SDKD log:137) "ViewName": "test_view"
[58.61 INFO] (SDKD log:137) }
[58.61 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.61 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[58.62 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.73 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.13
[58.73 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@106.14 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[58.73 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[58.74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.74 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[58.74 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.74 INFO] (SDKD log:137) "DS": {},
[58.74 INFO] (SDKD log:137) "ViewParameters": {},
[58.74 INFO] (SDKD log:137) "DesignName": "test_design",
[58.74 INFO] (SDKD log:137) "Options": {
[58.75 INFO] (SDKD log:137) "DelayMax": 10,
[58.75 INFO] (SDKD log:137) "DelayMin": 1,
[58.75 INFO] (SDKD log:137) "IterWait": 1,
[58.75 INFO] (SDKD log:137) "ViewQueryCount": -1,
[58.75 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[58.76 INFO] (SDKD log:137) "TimeRes": 1
[58.76 INFO] (SDKD log:137) },
[58.76 INFO] (SDKD log:137) "ViewName": "test_view"
[58.76 INFO] (SDKD log:137) }
[58.76 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.76 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[58.76 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.77 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.15
[58.77 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@107.16 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[58.78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[58.78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.78 INFO] (Workload call:112) Preloading items into cluster before workload
[58.78 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.78 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.17 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[58.79 INFO] (SDKD log:137) "DS": {},
[58.79 INFO] (SDKD log:137) "ViewParameters": {},
[58.79 INFO] (SDKD log:137) "DesignName": "test_design",
[58.79 INFO] (SDKD log:137) "Options": {
[58.79 INFO] (SDKD log:137) "DelayMax": 10,
[58.80 INFO] (SDKD log:137) "DelayMin": 1,
[58.80 INFO] (SDKD log:137) "IterWait": 1,
[58.80 INFO] (SDKD log:137) "ViewQueryCount": -1,
[58.80 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[58.80 INFO] (SDKD log:137) "TimeRes": 1
[58.81 INFO] (SDKD log:137) },
[58.81 INFO] (SDKD log:137) "ViewName": "test_view"
[58.81 INFO] (SDKD log:137) }
[58.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[58.81 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[58.82 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.17
[58.82 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.18 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=false, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=0, DelayMin=0, IterWait=100, TimeRes=0}}
[58.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[58.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[58.83 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[58.83 INFO] (SDKD log:137) "DS": {
[58.83 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[58.84 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[58.84 INFO] (SDKD log:137) "Count": 15000,
[58.84 INFO] (SDKD log:137) "Continuous": false,
[58.84 INFO] (SDKD log:137) "VSize": 128,
[58.84 INFO] (SDKD log:137) "Repeat": "REP",
[58.85 INFO] (SDKD log:137) "KSize": 32
[58.85 INFO] (SDKD log:137) },
[58.85 INFO] (SDKD log:137) "Options": {
[58.85 INFO] (SDKD log:137) "DelayMax": 0,
[58.85 INFO] (SDKD log:137) "DelayMin": 0,
[58.85 INFO] (SDKD log:137) "IterWait": 100,
[58.86 INFO] (SDKD log:137) "TimeRes": 0
[58.86 INFO] (SDKD log:137) }
[58.86 INFO] (SDKD log:137) }
[249.74 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@108.18 => {[OK]: 4003, [MEMD:MEMD_EVBUCKET]: 10997}
[249.74 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@108.19
[249.74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[249.74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[249.74 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[249.74 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[249.74 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.20 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.75 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.27 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.75 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.26 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.75 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.25 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[249.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.24 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.23 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.22 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.21 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.29 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.77 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.28 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.77 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[249.79 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[249.79 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.79 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.79 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[249.79 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.80 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[249.80 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.80 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[249.80 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.20
[249.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.81 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@109.30 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[249.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[249.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[249.81 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[249.82 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[249.82 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.31 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.82 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[249.82 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[249.82 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[249.82 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[249.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[249.83 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[249.83 INFO] (SDKD log:137) "DS": {
[249.83 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[249.83 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[249.83 INFO] (SDKD log:137) "Count": 15000,
[249.84 INFO] (SDKD log:137) "Continuous": true,
[249.84 INFO] (SDKD log:137) "VSize": 128,
[249.84 INFO] (SDKD log:137) "Repeat": "REP",
[249.84 INFO] (SDKD log:137) "KSize": 32
[249.84 INFO] (SDKD log:137) },
[249.84 INFO] (SDKD log:137) "Options": {
[249.84 INFO] (SDKD log:137) "DelayMax": 10,
[249.85 INFO] (SDKD log:137) "DelayMin": 1,
[249.85 INFO] (SDKD log:137) "IterWait": 1,
[249.85 INFO] (SDKD log:137) "TimeRes": 1
[249.85 INFO] (SDKD log:137) }
[249.86 INFO] (SDKD log:137) }
[249.86 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@114.25
[249.86 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.87 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@114.32 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[249.87 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[249.87 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[249.87 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[249.88 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.33 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.88 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[249.88 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[249.88 INFO] (SDKD log:137) "DS": {
[249.89 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[249.89 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[249.89 INFO] (SDKD log:137) "Count": 15000,
[249.89 INFO] (SDKD log:137) "Continuous": true,
[249.89 INFO] (SDKD log:137) "VSize": 128,
[249.90 INFO] (SDKD log:137) "Repeat": "REP",
[249.90 INFO] (SDKD log:137) "KSize": 32
[249.90 INFO] (SDKD log:137) },
[249.90 INFO] (SDKD log:137) "Options": {
[249.90 INFO] (SDKD log:137) "DelayMax": 10,
[249.90 INFO] (SDKD log:137) "DelayMin": 1,
[249.90 INFO] (SDKD log:137) "IterWait": 1,
[249.91 INFO] (SDKD log:137) "TimeRes": 1
[249.91 INFO] (SDKD log:137) }
[249.91 INFO] (SDKD log:137) }
[249.91 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.91 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@116.27
[249.92 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120
[249.92 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@116.34 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[249.92 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[249.93 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.35 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[249.93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[249.93 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[249.93 INFO] (SDKD log:137) "DS": {
[249.94 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[249.94 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[249.94 INFO] (SDKD log:137) "Count": 15000,
[249.94 INFO] (SDKD log:137) "Continuous": true,
[249.94 INFO] (SDKD log:137) "VSize": 128,
[249.94 INFO] (SDKD log:137) "Repeat": "REP",
[249.95 INFO] (SDKD log:137) "KSize": 32
[249.95 INFO] (SDKD log:137) },
[249.95 INFO] (SDKD log:137) "Options": {
[249.95 INFO] (SDKD log:137) "DelayMax": 10,
[249.95 INFO] (SDKD log:137) "DelayMin": 1,
[249.96 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@115.26
[249.96 INFO] (SDKD log:137) "IterWait": 1,
[249.96 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@115.36 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[249.96 INFO] (SDKD log:137) "TimeRes": 1
[249.96 INFO] (SDKD log:137) }
[249.96 INFO] (SDKD log:137) }
[249.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[249.97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@122.37 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[249.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121
[249.98 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[249.98 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[249.98 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[249.98 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[249.98 INFO] (SDKD log:137) "DS": {
[249.98 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[249.99 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[249.99 INFO] (SDKD log:137) "Count": 15000,
[249.99 INFO] (SDKD log:137) "Continuous": true,
[249.99 INFO] (SDKD log:137) "VSize": 128,
[249.99 INFO] (SDKD log:137) "Repeat": "REP",
[249.99 INFO] (SDKD log:137) "KSize": 32
[250.00 INFO] (SDKD log:137) },
[250.00 INFO] (SDKD log:137) "Options": {
[250.00 INFO] (SDKD log:137) "DelayMax": 10,
[250.01 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@113.24
[250.01 INFO] (SDKD log:137) "DelayMin": 1,
[250.01 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@113.38 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.01 INFO] (SDKD log:137) "IterWait": 1,
[250.02 INFO] (SDKD log:137) "TimeRes": 1
[250.02 INFO] (SDKD log:137) }
[250.02 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@123.39 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.02 INFO] (SDKD log:137) }
[250.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 122
[250.03 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.03 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.03 INFO] (SDKD log:137) "DS": {
[250.04 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.04 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.04 INFO] (SDKD log:137) "Count": 15000,
[250.04 INFO] (SDKD log:137) "Continuous": true,
[250.04 INFO] (SDKD log:137) "VSize": 128,
[250.04 INFO] (SDKD log:137) "Repeat": "REP",
[250.05 INFO] (SDKD log:137) "KSize": 32
[250.05 INFO] (SDKD log:137) },
[250.05 INFO] (SDKD log:137) "Options": {
[250.05 INFO] (SDKD log:137) "DelayMax": 10,
[250.05 INFO] (SDKD log:137) "DelayMin": 1,
[250.05 INFO] (SDKD log:137) "IterWait": 1,
[250.06 INFO] (SDKD log:137) "TimeRes": 1
[250.07 INFO] (SDKD log:137) }
[250.07 INFO] (SDKD log:137) }
[250.08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.08 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.21
[250.08 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@110.40 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.09 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 123
[250.09 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.09 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@124.41 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.10 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.11 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 124
[250.12 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.12 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.12 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.13 INFO] (SDKD log:137) "DS": {
[250.13 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[250.14 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.15 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.15 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[250.15 INFO] (SDKD log:137) "Count": 15000,
[250.15 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[250.15 INFO] (SDKD log:137) "Continuous": true,
[250.15 INFO] (SDKD log:137) "VSize": 128,
[250.16 INFO] (SDKD log:137) "Repeat": "REP",
[250.16 INFO] (SDKD log:137) "KSize": 32
[250.16 INFO] (SDKD log:137) },
[250.16 INFO] (SDKD log:137) "Options": {
[250.16 INFO] (SDKD log:137) "DelayMax": 10,
[250.17 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[250.17 INFO] (SDKD log:137) "DelayMin": 1,
[250.17 INFO] (SDKD log:137) "IterWait": 1,
[250.18 INFO] (SDKD log:137) at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Couchbase.IO.Strategies.EapConnection.<>c__DisplayClass3`1.<Send>b__0(FormatMessageHandler m) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line
38
   at Common.Logging.Factory.AbstractLogger.FormatMessageCallbackFormattedMessage.ToString()
   at log4net.ObjectRenderer.DefaultRenderer.RenderObject(RendererMap rendererMap, Object obj, TextWriter writer)
   at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj, TextWriter writer)
[250.18 INFO] (SDKD log:137) "TimeRes": 1
[250.18 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[250.19 INFO] (SDKD log:137) }
[250.19 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[250.19 INFO] (SDKD log:137) }
[250.19 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[250.20 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.21 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[250.21 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.22
[250.21 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@111.42 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.22 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[250.23 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@125.43 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.23 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.23 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.23 INFO] (SDKD log:137) "DS": {
[250.24 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.24 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.24 INFO] (SDKD log:137) "Count": 15000,
[250.24 INFO] (SDKD log:137) "Continuous": true,
[250.25 INFO] (SDKD log:137) "VSize": 128,
[250.25 INFO] (SDKD log:137) "Repeat": "REP",
[250.25 INFO] (SDKD log:137) "KSize": 32
[250.25 INFO] (SDKD log:137) },
[250.25 INFO] (SDKD log:137) "Options": {
[250.27 INFO] (SDKD log:137) "DelayMax": 10,
[250.28 INFO] (SDKD log:137) "DelayMin": 1,
[250.28 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@117.28
[250.28 INFO] (SDKD log:137) "IterWait": 1,
[250.29 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@117.44 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.29 INFO] (SDKD log:137) "TimeRes": 1
[250.29 INFO] (SDKD log:137) }
[250.29 INFO] (SDKD log:137) }
[250.30 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@126.45 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.30 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.31 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 125
[250.31 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.32 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.32 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.32 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.33 INFO] (SDKD log:137) "DS": {
[250.33 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.33 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.33 INFO] (SDKD log:137) "Count": 15000,
[250.34 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.23
[250.34 INFO] (SDKD log:137) "Continuous": true,
[250.35 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@112.46 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.35 INFO] (SDKD log:137) "VSize": 128,
[250.35 INFO] (SDKD log:137) "Repeat": "REP",
[250.35 INFO] (SDKD log:137) "KSize": 32
[250.35 INFO] (SDKD log:137) },
[250.36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@127.47 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.36 INFO] (SDKD log:137) "Options": {
[250.36 INFO] (SDKD log:137) "DelayMax": 10,
[250.36 INFO] (SDKD log:137) "DelayMin": 1,
[250.37 INFO] (SDKD log:137) "IterWait": 1,
[250.37 INFO] (SDKD log:137) "TimeRes": 1
[250.37 INFO] (SDKD log:137) }
[250.37 INFO] (SDKD log:137) }
[250.37 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.38 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 126
[250.38 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.38 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.38 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.38 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.38 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.39 INFO] (SDKD log:137) "DS": {
[250.39 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.40 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.40 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@118.29
[250.40 INFO] (SDKD log:137) "Count": 15000,
[250.40 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@118.48 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.41 INFO] (SDKD log:137) "Continuous": true,
[250.42 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@128.49 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[250.42 INFO] (SDKD log:137) "VSize": 128,
[250.43 INFO] (SDKD log:137) "Repeat": "REP",
[250.43 INFO] (SDKD log:137) "KSize": 32
[250.43 INFO] (SDKD log:137) },
[250.43 INFO] (SDKD log:137) "Options": {
[250.44 INFO] (SDKD log:137) "DelayMax": 10,
[250.44 INFO] (SDKD log:137) "DelayMin": 1,
[250.44 INFO] (SDKD log:137) "IterWait": 1,
[250.44 INFO] (SDKD log:137) "TimeRes": 1
[250.44 INFO] (SDKD log:137) }
[250.45 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@119.31
[250.45 INFO] (SDKD log:137) }
[250.45 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 127
[250.45 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@119.50 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.46 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.46 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[250.46 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.47 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.47 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.47 INFO] (SDKD log:137) "DS": {
[250.47 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.48 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.48 INFO] (SDKD log:137) "Count": 15000,
[250.48 INFO] (SDKD log:137) "Continuous": true,
[250.48 INFO] (SDKD log:137) "VSize": 128,
[250.49 INFO] (SDKD log:137) "Repeat": "REP",
[250.49 INFO] (SDKD log:137) "KSize": 32
[250.49 INFO] (SDKD log:137) },
[250.49 INFO] (SDKD log:137) "Options": {
[250.50 INFO] (SDKD log:137) "DelayMax": 10,
[250.50 INFO] (SDKD log:137) "DelayMin": 1,
[250.50 INFO] (SDKD log:137) "IterWait": 1,
[250.50 INFO] (SDKD log:137) "TimeRes": 1
[250.51 INFO] (SDKD log:137) }
[250.51 INFO] (SDKD log:137) }
[250.51 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 128
[250.51 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@120.33
[250.52 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@120.51 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.52 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@121.35
[250.56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.56 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@121.52 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.56 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.56 INFO] (SDKD log:137) "DS": {
[250.57 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.57 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.57 INFO] (SDKD log:137) "Count": 15000,
[250.58 INFO] (SDKD log:137) "Continuous": true,
[250.58 INFO] (SDKD log:137) "VSize": 128,
[250.58 INFO] (SDKD log:137) "Repeat": "REP",
[250.58 INFO] (SDKD log:137) "KSize": 32
[250.59 INFO] (SDKD log:137) },
[250.59 INFO] (SDKD log:137) "Options": {
[250.59 INFO] (SDKD log:137) "DelayMax": 10,
[250.59 INFO] (SDKD log:137) +++ Received exception of ID 0
[250.60 INFO] (SDKD log:137) "DelayMin": 1,
[250.60 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[250.60 INFO] (SDKD log:137) "IterWait": 1,
[250.60 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[250.60 INFO] (SDKD log:137) "TimeRes": 1
[250.60 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[250.60 INFO] (SDKD log:137) }
[250.61 INFO] (SDKD log:137) }
[250.61 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.61 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[250.62 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.62 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.62 INFO] (SDKD log:137) "DS": {
[250.63 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.63 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.63 INFO] (SDKD log:137) "Count": 15000,
[250.63 INFO] (SDKD log:137) "Continuous": true,
[250.63 INFO] (SDKD log:137) "VSize": 128,
[250.63 INFO] (SDKD log:137) "Repeat": "REP",
[250.63 INFO] (SDKD log:137) "KSize": 32
[250.63 INFO] (SDKD log:137) },
[250.64 INFO] (SDKD log:137) "Options": {
[250.64 INFO] (SDKD log:137) "DelayMax": 10,
[250.64 INFO] (SDKD log:137) "DelayMin": 1,
[250.64 INFO] (SDKD log:137) "IterWait": 1,
[250.64 INFO] (SDKD log:137) "TimeRes": 1
[250.65 INFO] (SDKD log:137) }
[250.65 INFO] (SDKD log:137) }
[250.65 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.65 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.65 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.65 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.66 INFO] (SDKD log:137) "DS": {
[250.66 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.66 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.66 INFO] (SDKD log:137) "Count": 15000,
[250.66 INFO] (SDKD log:137) "Continuous": true,
[250.66 INFO] (SDKD log:137) "VSize": 128,
[250.67 INFO] (SDKD log:137) "Repeat": "REP",
[250.67 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@122.37
[250.67 INFO] (SDKD log:137) "KSize": 32
[250.68 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@122.53 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.68 INFO] (SDKD log:137) },
[250.68 INFO] (SDKD log:137) "Options": {
[250.68 INFO] (SDKD log:137) "DelayMax": 10,
[250.68 INFO] (SDKD log:137) "DelayMin": 1,
[250.69 INFO] (SDKD log:137) "IterWait": 1,
[250.69 INFO] (SDKD log:137) "TimeRes": 1
[250.69 INFO] (SDKD log:137) }
[250.70 INFO] (SDKD log:137) }
[250.70 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.70 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[250.71 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.71 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.71 INFO] (SDKD log:137) "DS": {
[250.71 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@123.39
[250.71 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.72 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@123.54 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.72 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.73 INFO] (SDKD log:137) "Count": 15000,
[250.73 INFO] (SDKD log:137) "Continuous": true,
[250.73 INFO] (SDKD log:137) "VSize": 128,
[250.73 INFO] (SDKD log:137) "Repeat": "REP",
[250.73 INFO] (SDKD log:137) "KSize": 32
[250.73 INFO] (SDKD log:137) },
[250.74 INFO] (SDKD log:137) "Options": {
[250.74 INFO] (SDKD log:137) "DelayMax": 10,
[250.74 INFO] (SDKD log:137) "DelayMin": 1,
[250.75 INFO] (SDKD log:137) "IterWait": 1,
[250.75 INFO] (SDKD log:137) "TimeRes": 1
[250.75 INFO] (SDKD log:137) }
[250.76 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@124.41
[250.76 INFO] (SDKD log:137) }
[250.76 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.76 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@124.55 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.77 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.77 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.77 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.78 INFO] (SDKD log:137) "DS": {
[250.78 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.78 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.79 INFO] (SDKD log:137) "Count": 15000,
[250.79 INFO] (SDKD log:137) "Continuous": true,
[250.79 INFO] (SDKD log:137) "VSize": 128,
[250.79 INFO] (SDKD log:137) "Repeat": "REP",
[250.79 INFO] (SDKD log:137) "KSize": 32
[250.79 INFO] (SDKD log:137) },
[250.79 INFO] (SDKD log:137) "Options": {
[250.80 INFO] (SDKD log:137) "DelayMax": 10,
[250.80 INFO] (SDKD log:137) "DelayMin": 1,
[250.80 INFO] (SDKD log:137) "IterWait": 1,
[250.81 INFO] (SDKD log:137) "TimeRes": 1
[250.81 INFO] (SDKD log:137) }
[250.81 INFO] (SDKD log:137) }
[250.81 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@125.43
[250.82 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.83 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@125.56 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[250.84 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.84 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@126.45
[250.85 INFO] (SDKD log:137) "DS": {
[250.85 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.85 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@126.57 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.86 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.86 INFO] (SDKD log:137) "Count": 15000,
[250.87 INFO] (SDKD log:137) "Continuous": true,
[250.87 INFO] (SDKD log:137) "VSize": 128,
[250.87 INFO] (SDKD log:137) "Repeat": "REP",
[250.87 INFO] (SDKD log:137) "KSize": 32
[250.87 INFO] (SDKD log:137) },
[250.87 INFO] (SDKD log:137) "Options": {
[250.88 INFO] (SDKD log:137) "DelayMax": 10,
[250.88 INFO] (SDKD log:137) "DelayMin": 1,
[250.88 INFO] (SDKD log:137) "IterWait": 1,
[250.88 INFO] (SDKD log:137) "TimeRes": 1
[250.88 INFO] (SDKD log:137) }
[250.88 INFO] (SDKD log:137) }
[250.89 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.89 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.89 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.89 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.89 INFO] (SDKD log:137) "DS": {
[250.90 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.90 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.90 INFO] (SDKD log:137) "Count": 15000,
[250.90 INFO] (SDKD log:137) "Continuous": true,
[250.91 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@127.47
[250.91 INFO] (SDKD log:137) "VSize": 128,
[250.91 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@127.58 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[250.91 INFO] (SDKD log:137) "Repeat": "REP",
[250.91 INFO] (SDKD log:137) "KSize": 32
[250.92 INFO] (SDKD log:137) },
[250.92 INFO] (SDKD log:137) "Options": {
[250.92 INFO] (SDKD log:137) "DelayMax": 10,
[250.93 INFO] (SDKD log:137) "DelayMin": 1,
[250.93 INFO] (SDKD log:137) "IterWait": 1,
[250.93 INFO] (SDKD log:137) "TimeRes": 1
[250.93 INFO] (SDKD log:137) }
[250.93 INFO] (SDKD log:137) }
[250.93 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.94 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[250.94 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.94 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.94 INFO] (SDKD log:137) "DS": {
[250.94 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.95 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.95 INFO] (SDKD log:137) "Count": 15000,
[250.96 INFO] (SDKD log:137) "Continuous": true,
[250.96 INFO] (SDKD log:137) "VSize": 128,
[250.96 INFO] (SDKD log:137) "Repeat": "REP",
[250.96 INFO] (SDKD log:137) "KSize": 32
[250.96 INFO] (SDKD log:137) },
[250.96 INFO] (SDKD log:137) "Options": {
[250.96 INFO] (SDKD log:137) "DelayMax": 10,
[250.96 INFO] (SDKD log:137) "DelayMin": 1,
[250.96 INFO] (SDKD log:137) "IterWait": 1,
[250.96 INFO] (SDKD log:137) "TimeRes": 1
[250.96 INFO] (SDKD log:137) }
[250.97 INFO] (SDKD log:137) }
[250.97 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[250.97 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[250.97 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[250.98 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[250.98 INFO] (SDKD log:137) "DS": {
[250.99 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[250.99 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[250.99 INFO] (SDKD log:137) "Count": 15000,
[250.99 INFO] (SDKD log:137) "Continuous": true,
[250.99 INFO] (SDKD log:137) "VSize": 128,
[250.99 INFO] (SDKD log:137) "Repeat": "REP",
[250.99 INFO] (SDKD log:137) "KSize": 32
[250.99 INFO] (SDKD log:137) },
[251.00 INFO] (SDKD log:137) "Options": {
[251.00 INFO] (SDKD log:137) "DelayMax": 10,
[251.00 INFO] (SDKD log:137) "DelayMin": 1,
[251.00 INFO] (SDKD log:137) "IterWait": 1,
[251.01 INFO] (SDKD log:137) "TimeRes": 1
[251.01 INFO] (SDKD log:137) }
[251.02 INFO] (SDKD log:137) }
[251.05 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@128.49
[251.06 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@128.59 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[251.07 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@129.60 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[251.08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[251.09 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[251.09 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[251.09 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[251.09 INFO] (SDKD log:137) +++ Received exception of ID 0
[251.09 INFO] (SDKD log:137) "DS": {
[251.09 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[251.09 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[251.09 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[251.10 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[251.10 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[251.10 INFO] (SDKD log:137) "Count": 15000,
[251.10 INFO] (SDKD log:137) +++ Received exception of ID 0
[251.10 INFO] (SDKD log:137) "Continuous": true,
[251.11 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[251.11 INFO] (SDKD log:137) "VSize": 128,
[251.11 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[251.11 INFO] (SDKD log:137) "Repeat": "REP",
[251.12 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[251.13 INFO] (SDKD log:137) "KSize": 32
[251.13 INFO] (SDKD log:137) },
[251.13 INFO] (SDKD log:137) "Options": {
[251.13 INFO] (SDKD log:137) "DelayMax": 10,
[251.14 INFO] (SDKD log:137) "DelayMin": 1,
[251.14 INFO] (SDKD log:137) "IterWait": 1,
[251.14 INFO] (SDKD log:137) "TimeRes": 1
[251.15 INFO] (SDKD log:137) }
[251.15 INFO] (SDKD log:137) }
[251.15 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 129
[251.15 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[251.17 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@129.60
[251.18 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@129.61 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=
12}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[251.19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[251.20 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[251.21 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[251.21 INFO] (SDKD log:137) "DS": {
[251.21 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[251.21 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[251.21 INFO] (SDKD log:137) "Count": 1000,
[251.22 INFO] (SDKD log:137) "Continuous": false,
[251.22 INFO] (SDKD log:137) "VSize": 512,
[251.23 INFO] (SDKD log:137) "Repeat": "rep",
[251.23 INFO] (SDKD log:137) "KSize": 12
[251.23 INFO] (SDKD log:137) },
[251.23 INFO] (SDKD log:137) "Schema": {
[251.24 INFO] (SDKD log:137) "InflateContent": "meh",
[251.24 INFO] (SDKD log:137) "InflateLevel": 40
[251.24 INFO] (SDKD log:137) },
[251.25 INFO] (SDKD log:137) "Options": {
[251.25 INFO] (SDKD log:137) "DelayMax": 0,
[251.25 INFO] (SDKD log:137) "DelayMin": 0,
[251.26 INFO] (SDKD log:137) "IterWait": 1,
[251.26 INFO] (SDKD log:137) "TimeRes": 0
[251.27 INFO] (SDKD log:137) }
[251.27 INFO] (SDKD log:137) }
[544.61 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[544.61 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@129.61
[544.61 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@129.62
[544.62 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[544.62 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@130.63 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[544.62 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[544.62 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[544.62 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[544.63 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[544.63 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[544.63 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 130
[544.63 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[544.67 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@130.63
[544.67 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@130.64 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[544.67 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[544.68 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@131.65 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[544.68 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[544.68 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[544.68 INFO] (SDKD log:137) "DS": {},
[544.68 INFO] (SDKD log:137) "ViewParameters": {
[544.69 INFO] (SDKD log:137) "limit": 0
[544.69 INFO] (SDKD log:137) },
[544.69 INFO] (SDKD log:137) "DesignName": "test_design",
[544.69 INFO] (SDKD log:137) "Options": {
[544.69 INFO] (SDKD log:137) "DelayMax": 10,
[544.70 INFO] (SDKD log:137) "DelayMin": 1,
[544.70 INFO] (SDKD log:137) "IterWait": 1,
[544.70 INFO] (SDKD log:137) "ViewQueryCount": -1,
[544.70 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[544.70 INFO] (SDKD log:137) "TimeRes": 1
[544.70 INFO] (SDKD log:137) },
[544.70 INFO] (SDKD log:137) "ViewName": "test_view"
[544.71 INFO] (SDKD log:137) }
[544.71 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[544.71 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 131
[544.71 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[544.73 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@131.65
[544.73 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@131.66 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[544.73 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[544.74 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@132.67 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[544.74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[544.74 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[544.74 INFO] (SDKD log:137) "DS": {},
[544.74 INFO] (SDKD log:137) "ViewParameters": {
[544.74 INFO] (SDKD log:137) "limit": 0
[544.75 INFO] (SDKD log:137) },
[544.75 INFO] (SDKD log:137) "DesignName": "test_design",
[544.75 INFO] (SDKD log:137) "Options": {
[544.75 INFO] (SDKD log:137) "DelayMax": 10,
[544.75 INFO] (SDKD log:137) "DelayMin": 1,
[544.76 INFO] (SDKD log:137) "IterWait": 1,
[544.76 INFO] (SDKD log:137) "ViewQueryCount": -1,
[544.76 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[544.76 INFO] (SDKD log:137) "TimeRes": 1
[544.76 INFO] (SDKD log:137) },
[544.77 INFO] (SDKD log:137) "ViewName": "test_view"
[544.77 INFO] (SDKD log:137) +++ Received exception of ID 0
[544.77 INFO] (SDKD log:137) }
[544.78 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[544.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[544.78 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[544.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 132
[544.78 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[544.79 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[544.81 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@132.67
[544.82 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@132.68 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[544.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[544.83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@133.69 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[544.83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[544.83 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[544.84 INFO] (SDKD log:137) "DS": {},
[544.84 INFO] (SDKD log:137) "ViewParameters": {
[544.85 INFO] (SDKD log:137) "limit": 0
[544.85 INFO] (SDKD log:137) },
[544.85 INFO] (SDKD log:137) "DesignName": "test_design",
[544.85 INFO] (SDKD log:137) "Options": {
[544.86 INFO] (SDKD log:137) "DelayMax": 10,
[544.86 INFO] (SDKD log:137) "DelayMin": 1,
[544.86 INFO] (SDKD log:137) "IterWait": 1,
[544.86 INFO] (SDKD log:137) "ViewQueryCount": -1,
[544.86 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[544.86 INFO] (SDKD log:137) +++ Received exception of ID 0
[544.87 INFO] (SDKD log:137) "TimeRes": 1
[544.87 INFO] (SDKD log:137) },
[544.88 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[544.88 INFO] (SDKD log:137) "ViewName": "test_view"
[544.88 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[544.88 INFO] (SDKD log:137) }
[544.88 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[544.89 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[544.89 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 133
[544.89 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[544.90 INFO] (SDKD log:137) +++ Received exception of ID 0
[544.91 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[544.92 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[544.92 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[544.93 INFO] (SDKD log:137) +++ Received exception of ID 0
[544.94 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[544.94 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[544.94 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[544.96 INFO] (SDKD log:137) +++ Received exception of ID 0
[544.97 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[544.97 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[544.97 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[544.97 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.00 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.00 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.02 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@133.69
[545.02 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.02 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@133.70 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[545.02 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[545.03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@134.71 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[545.03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[545.03 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[545.03 INFO] (SDKD log:137) "DS": {},
[545.03 INFO] (SDKD log:137) "ViewParameters": {
[545.03 INFO] (SDKD log:137) "limit": 0
[545.03 INFO] (SDKD log:137) },
[545.04 INFO] (SDKD log:137) "DesignName": "test_design",
[545.04 INFO] (SDKD log:137) "Options": {
[545.04 INFO] (SDKD log:137) "DelayMax": 10,
[545.04 INFO] (SDKD log:137) "DelayMin": 1,
[545.04 INFO] (SDKD log:137) "IterWait": 1,
[545.04 INFO] (SDKD log:137) "ViewQueryCount": -1,
[545.05 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[545.05 INFO] (SDKD log:137) "TimeRes": 1
[545.05 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.05 INFO] (SDKD log:137) },
[545.05 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.06 INFO] (SDKD log:137) "ViewName": "test_view"
[545.06 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.06 INFO] (SDKD log:137) }
[545.06 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.06 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[545.06 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.07 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 134
[545.07 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.07 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[545.07 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.08 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.08 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.08 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.08 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.09 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.09 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.10 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.10 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.10 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.10 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.10 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.11 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.11 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[545.13 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@134.71
[545.13 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@134.72 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[545.14 INFO] (Scenario run:83) RAMP phase started. Waiting for 30 seconds
[545.14 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[545.15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[545.15 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[545.15 INFO] (SDKD log:137) "DS": {},
[545.15 INFO] (SDKD log:137) "ViewParameters": {
[545.15 INFO] (SDKD log:137) "limit": 0
[545.15 INFO] (SDKD log:137) },
[545.16 INFO] (SDKD log:137) "DesignName": "test_design",
[545.16 INFO] (SDKD log:137) "Options": {
[545.16 INFO] (SDKD log:137) "DelayMax": 10,
[545.16 INFO] (SDKD log:137) "DelayMin": 1,
[545.16 INFO] (SDKD log:137) "IterWait": 1,
[545.16 INFO] (SDKD log:137) "ViewQueryCount": -1,
[545.17 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[545.17 INFO] (SDKD log:137) "TimeRes": 1
[545.17 INFO] (SDKD log:137) },
[545.17 INFO] (SDKD log:137) "ViewName": "test_view"
[545.18 INFO] (SDKD log:137) +++ Received exception of ID 0
[545.18 INFO] (SDKD log:137) }
[545.18 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[545.18 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[545.18 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[575.14 INFO] (Scenario run:86) Starting CHANGE phase
[575.20 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[576.22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[577.24 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[579.99 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[581.33 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[583.27 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 1%
[586.30 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[587.51 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[588.55 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3%
[589.56 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3%
[590.57 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3%
[591.63 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 4%
[592.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 5%
[593.81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 5%
[594.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 6%
[595.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 7%
[596.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 7%
[597.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 8%
[598.92 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 9%
[599.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 9%
[600.94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10%
[601.98 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10%
[603.05 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 11%
[604.06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 11%
[605.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 12%
[606.14 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 12%
[607.16 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 13%
[608.17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 13%
[609.28 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 14%
[610.34 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 14%
[611.35 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15%
[612.39 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15%
[613.41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 16%
[614.42 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 16%
[615.43 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[616.45 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[617.49 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[618.50 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 19%
[619.52 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 19%
[620.53 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 20%
[621.55 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 20%
[622.56 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 21%
[623.63 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 21%
[624.66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 22%
[625.66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 22%
[626.71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 23%
[627.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 24%
[628.77 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 24%
[629.79 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 25%
[630.80 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 25%
[631.81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 26%
[632.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 26%
[633.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 26%
[634.87 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 27%
[635.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 27%
[636.90 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 28%
[637.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 28%
[638.92 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 29%
[639.94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 29%
[640.95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 30%
[641.99 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 31%
[643.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 31%
[643.63 INFO] (SDKD log:137) +++ Received exception of ID 0
[643.63 INFO] (SDKD log:137)
[643.66 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
[644.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32%
[645.04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32%
[646.05 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32%
[647.06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 33%
[648.06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 33%
[649.07 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 34%
[650.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 35%
[651.17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 35%
[652.18 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 36%
[653.22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 36%
[654.22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 37%
[655.23 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 37%
[656.24 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 38%
[657.25 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 39%
[658.30 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 40%
[659.34 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 40%
[660.35 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 40%
[661.36 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 41%
[662.36 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 42%
[663.37 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 42%
[664.38 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 43%
[665.40 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 43%
[666.41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 44%
[667.42 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 45%
[668.43 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 45%
[669.45 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 46%
[670.47 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 47%
[671.52 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 47%
[672.52 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 48%
[673.54 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 48%
[674.55 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 49%
[675.58 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 49%
[676.58 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 50%
[677.59 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 50%
[678.60 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 51%
[679.60 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 51%
[680.61 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 51%
[681.62 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 52%
[682.63 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 53%
[683.64 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 53%
[684.64 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 54%
[685.65 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 54%
[686.65 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 55%
[687.66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 55%
[688.67 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[689.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[690.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[691.69 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 57%
[692.70 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 57%
[693.70 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 58%
[694.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 58%
[695.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 59%
[696.70 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[696.71 INFO] (SDKD log:137) at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at System.Collections.Generic.List`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
   at Couchbase.Configuration.CouchbaseConfigContext.LoadConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\CouchbaseConfigContext.cs:line
 47
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.UpdateConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Co
nfiguration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 91
   at Couchbase.Core.ClusterManager.NotifyConfigPublished(IBucketConfig bucketConfig) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 137
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.c
s:line 234
   at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, IServer server) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couc
hbase\Core\Buckets\CouchbaseBucket.cs:line 166
   at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 104
   at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 888
   at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\GetCommand.cs:line 18
   at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[696.74 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
        at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
[696.76 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 59%
[696.76 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.73
[696.76 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]

c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>

 Comments   
Comment by Jeff Morris [ 18/Aug/14 ]
This is valid bug. Threads 26 and 48 have both received a new config and updating the config context at the same time:

Line 1398139: 2014-08-16 08:55:03,007 [26] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#2061 on thread 26
Line 1398143: 2014-08-16 08:55:03,013 [48] INFO Couchbase.IO.Operations.OperationResult`1 - Received config rev#2061
Line 1398145: 2014-08-16 08:55:03,014 [48] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#2061 on thread 48
Line 1398175: 2014-08-16 08:55:03,032 [48] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - New config has changed new Rev#2062 | old Rev#2061 CCCP: {"name":"default","bucketType":null,"authType":null"...
Line 1398367: 2014-08-16 08:55:56,152 [26] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - New config has changed new Rev#2062 | old Rev#2061 CCCP:

Since they are both updating the same List<T>, the enumeration is failing (you cannot modify a collection once an enumeration has begin). This should be relatively easy to fix with some thread sync techniques.
Comment by Jeff Morris [ 19/Aug/14 ]
Patch:

git fetch ssh://jmorris@review.couchbase.org:29418/couchbase-net-client refs/changes/15/40715/1 && git checkout FETCH_HEAD
Comment by Andrei Baranouski [ 21/Aug/14 ]
fixed




[NCBC-592] AuthenticationException in CouchbaseBucket.SendWithRetry Created: 20/Aug/14  Updated: 21/Aug/14  Resolved: 20/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-592.log.zip    

 Description   
git log
commit 3fefdd5a17884c01d5113d73db2160bec9eb55db
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 18 22:31:37 2014 -0700

    NCBC-589: ThrowInvalidOperationException in CouchbaseConfigContext.LoadConfig during rebalance

    Change-Id: I813092f07bb6fde414ea4e05a67239173a540568

commit 732f2dcb72fe53cf7fbce27ac631a302ad9fe178
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 18 22:15:49 2014 -0700

    Check that length is positive in TypeSerializer.Deserialize

    Change-Id: Ic229fbdd0362593ac7577350cc04f549380ac7f5

commit 5c2c0e3589b5f0099fc8048de8b398c28f6eb04c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Tue Aug 12 16:48:03 2014 -0700

    NCBC-581: System.InvalidOperationException when server list is empty

    Change-Id: I8fdef46b4057bc9f3b898b789fcccd78f689fdb6
    Reviewed-on: http://review.couchbase.org/40546
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit a132c5f921dcd9aed04d8f0d55dc5ba8ad36dd8e



C:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,14 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/count=2
    workload=HybridWorkloadGroup
    rebalance/mode=out
    testcase=RebalanceScenario
    Logging to C:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-20-14\025327\b0efa6fcc6ef738b9ac6a2109416adad

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-count 2 --workload HybridWorkloadGroup
 \ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1,32 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1,61 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,63 INFO] (RunContext run:124) Starting cluster and driver
[1,63 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1,63 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[1,64 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,93 INFO] (SDKD log:137) --- Logging Self-Test ---
[1,93 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[1,93 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[1,94 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[1,94 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[1,94 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,19 INFO] (SDKD log:137) Call SweepOnce()
[2,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,20 INFO] (SDKD log:137) Call SweepOnce()
[2,20 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,21 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[2,32 INFO] (SDKD log:137) Call SweepOnce()
[2,32 INFO] (SDKD log:137) Call SweepOnce()
[2,34 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[8,62 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.145:8091,ns_1@10.3.4.145>
[9,08 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.146:8091,ns_1@10.3.4.146>
[9,52 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[10,10 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[10,10 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,11 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,11 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[12,30 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[12,31 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[12,31 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[12,32 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[12,36 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[12,36 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[12,36 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[12,36 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[13,07 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@5417c2f
[13,07 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@65606a8d
[13,07 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@6653fd5c
[13,08 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@c890d48
[13,08 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@994c3c9
[33,08 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[39,35 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[52,86 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[55,92 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[56,79 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[56,79 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[57,01 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[57,24 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[66,46 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[69,00 INFO] (RunContext run:143) Driver and cluster initialized
[69,25 INFO] (RunContext call:167) Running scenario..
[69,25 INFO] (Scenario run:72) Starting RAMP phase
[69,25 INFO] (Workload setupDesign:63) Creating design test_design
[70,38 INFO] (Workload setupDesign:80) Design creation done
[70,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[70,39 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[70,39 INFO] (SDKD log:137) Call SweepOnce()
[70,39 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[70,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[70,42 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[70,43 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[70,45 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[70,46 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[78,02 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[78,02 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {Options={DelayMin=0, TimeRes=0, IterWait=1, DelayMax=0}, Schema={InflateLevel=40, InflateContent=meh}, DSType=DSTYPE_SEEDED, DS={KSize=12, KSeed=ViewFillerSeed, VSize=512, VSeed=Vie
wFillerSeed, Continuous=false, Repeat=rep, Count=1000}}
[78,03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[78,03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[78,03 INFO] (SDKD log:137) "Options": {
[78,03 INFO] (SDKD log:137) "DelayMin": 0,
[78,03 INFO] (SDKD log:137) "TimeRes": 0,
[78,04 INFO] (SDKD log:137) "IterWait": 1,
[78,04 INFO] (SDKD log:137) "DelayMax": 0
[78,04 INFO] (SDKD log:137) },
[78,05 INFO] (SDKD log:137) "Schema": {
[78,05 INFO] (SDKD log:137) "InflateLevel": 40,
[78,05 INFO] (SDKD log:137) "InflateContent": "meh"
[78,05 INFO] (SDKD log:137) },
[78,05 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[78,06 INFO] (SDKD log:137) "DS": {
[78,06 INFO] (SDKD log:137) "KSize": 12,
[78,07 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[78,07 INFO] (SDKD log:137) "VSize": 512,
[78,07 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[78,08 INFO] (SDKD log:137) "Continuous": false,
[78,08 INFO] (SDKD log:137) "Repeat": "rep",
[78,08 INFO] (SDKD log:137) "Count": 1000
[78,08 INFO] (SDKD log:137) }
[78,09 INFO] (SDKD log:137) }
[230,25 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[230,25 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[230,26 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[230,26 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[230,26 INFO] (Workload call:112) Preloading items into cluster before workload
[230,26 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[230,27 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[230,27 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[230,27 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[230,27 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[230,27 INFO] (SDKD log:137) Call SweepOnce()
[230,28 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[230,28 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[230,29 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[237,18 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[237,18 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@102.5 => {Options={DelayMin=0, TimeRes=0, IterWait=100, DelayMax=0}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=false, Repeat=REP, Coun
t=15000}}
[237,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[237,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[237,20 INFO] (SDKD log:137) "Options": {
[237,20 INFO] (SDKD log:137) "DelayMin": 0,
[237,20 INFO] (SDKD log:137) "TimeRes": 0,
[237,20 INFO] (SDKD log:137) "IterWait": 100,
[237,21 INFO] (SDKD log:137) "DelayMax": 0
[237,21 INFO] (SDKD log:137) },
[237,21 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[237,21 INFO] (SDKD log:137) "DS": {
[237,22 INFO] (SDKD log:137) "KSize": 32,
[237,22 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[237,23 INFO] (SDKD log:137) "VSize": 128,
[237,23 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[237,23 INFO] (SDKD log:137) "Continuous": false,
[237,24 INFO] (SDKD log:137) "Repeat": "REP",
[237,24 INFO] (SDKD log:137) "Count": 15000
[237,24 INFO] (SDKD log:137) }
[237,24 INFO] (SDKD log:137) }
[412,79 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[1734,81 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@102.5 => {[OK]: 14990, [MEMD:GENERIC]: 10}
[1734,81 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[1734,81 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[1734,81 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[1734,81 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1508015768
[1734,82 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1508015768
[1734,82 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.11 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,82 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.16 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,82 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.12 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.14 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.15 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.13 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.10 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.9 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.8 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1734,83 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,83 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,84 INFO] (SDKD log:137) Call SweepOnce()
[1734,84 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,84 INFO] (SDKD log:137) Call SweepOnce()
[1734,84 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,84 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,85 INFO] (SDKD log:137) Call SweepOnce()
[1734,85 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,86 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,86 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,86 INFO] (SDKD log:137) Call SweepOnce()
[1734,86 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,87 INFO] (SDKD log:137) Call SweepOnce()
[1734,87 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,87 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,88 INFO] (SDKD log:137) Call SweepOnce()
[1734,88 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,88 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,89 INFO] (SDKD log:137) Call SweepOnce()
[1734,89 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,89 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,89 INFO] (SDKD log:137) Call SweepOnce()
[1734,89 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[1734,91 INFO] (SDKD log:137) Call SweepOnce()
[1734,91 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,91 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1734,91 INFO] (SDKD log:137) Call SweepOnce()
[1734,92 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1734,92 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[1734,92 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[1734,93 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[1734,93 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[1734,93 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[1734,94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[1734,94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[1734,94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[1734,94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[1741,05 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.11
[1741,10 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@107.17 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1741,10 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1741,11 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1741,11 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.18 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1741,11 INFO] (SDKD log:137) "Options": {
[1741,11 INFO] (SDKD log:137) "DelayMin": 1,
[1741,11 INFO] (SDKD log:137) "TimeRes": 1,
[1741,12 INFO] (SDKD log:137) "IterWait": 1,
[1741,12 INFO] (SDKD log:137) "DelayMax": 10
[1741,13 INFO] (SDKD log:137) },
[1741,13 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1741,13 INFO] (SDKD log:137) "DS": {
[1741,13 INFO] (SDKD log:137) "KSize": 32,
[1741,13 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1741,14 INFO] (SDKD log:137) "VSize": 128,
[1741,14 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1741,15 INFO] (SDKD log:137) "Continuous": true,
[1741,15 INFO] (SDKD log:137) "Repeat": "REP",
[1741,15 INFO] (SDKD log:137) "Count": 15000
[1741,15 INFO] (SDKD log:137) }
[1741,16 INFO] (SDKD log:137) }
[1741,16 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1741,16 INFO] (SDKD log:137) Call SweepOnce()
[1741,16 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1741,17 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[1746,76 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.16
[1746,76 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@112.19 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1746,76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1746,76 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.20 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1746,76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1746,77 INFO] (SDKD log:137) "Options": {
[1746,77 INFO] (SDKD log:137) "DelayMin": 1,
[1746,77 INFO] (SDKD log:137) "TimeRes": 1,
[1746,77 INFO] (SDKD log:137) "IterWait": 1,
[1746,78 INFO] (SDKD log:137) "DelayMax": 10
[1746,78 INFO] (SDKD log:137) },
[1746,78 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1746,79 INFO] (SDKD log:137) "DS": {
[1746,79 INFO] (SDKD log:137) "KSize": 32,
[1746,79 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1746,80 INFO] (SDKD log:137) "VSize": 128,
[1746,80 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1746,80 INFO] (SDKD log:137) "Continuous": true,
[1746,80 INFO] (SDKD log:137) "Repeat": "REP",
[1746,80 INFO] (SDKD log:137) "Count": 15000
[1746,81 INFO] (SDKD log:137) }
[1746,81 INFO] (SDKD log:137) }
[1746,81 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1746,82 INFO] (SDKD log:137) Call SweepOnce()
[1746,82 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1746,82 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[1752,28 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.12
[1752,28 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.21 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1752,28 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1752,28 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1752,29 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.22 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1752,29 INFO] (SDKD log:137) "Options": {
[1752,29 INFO] (SDKD log:137) "DelayMin": 1,
[1752,29 INFO] (SDKD log:137) "TimeRes": 1,
[1752,29 INFO] (SDKD log:137) "IterWait": 1,
[1752,30 INFO] (SDKD log:137) "DelayMax": 10
[1752,30 INFO] (SDKD log:137) },
[1752,30 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1752,31 INFO] (SDKD log:137) "DS": {
[1752,31 INFO] (SDKD log:137) "KSize": 32,
[1752,31 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1752,31 INFO] (SDKD log:137) "VSize": 128,
[1752,32 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1752,32 INFO] (SDKD log:137) "Continuous": true,
[1752,32 INFO] (SDKD log:137) "Repeat": "REP",
[1752,33 INFO] (SDKD log:137) "Count": 15000
[1752,33 INFO] (SDKD log:137) }
[1752,33 INFO] (SDKD log:137) }
[1752,34 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1752,34 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1752,34 INFO] (SDKD log:137) Call SweepOnce()
[1752,34 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[1766,32 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.14
[1766,32 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@110.23 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1766,32 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1766,33 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.24 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1766,33 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1766,33 INFO] (SDKD log:137) "Options": {
[1766,33 INFO] (SDKD log:137) "DelayMin": 1,
[1766,33 INFO] (SDKD log:137) "TimeRes": 1,
[1766,34 INFO] (SDKD log:137) "IterWait": 1,
[1766,34 INFO] (SDKD log:137) "DelayMax": 10
[1766,34 INFO] (SDKD log:137) },
[1766,35 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1766,35 INFO] (SDKD log:137) "DS": {
[1766,35 INFO] (SDKD log:137) "KSize": 32,
[1766,35 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1766,36 INFO] (SDKD log:137) "VSize": 128,
[1766,36 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1766,36 INFO] (SDKD log:137) "Continuous": true,
[1766,37 INFO] (SDKD log:137) "Repeat": "REP",
[1766,37 INFO] (SDKD log:137) "Count": 15000
[1766,37 INFO] (SDKD log:137) }
[1766,37 INFO] (SDKD log:137) }
[1766,37 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1766,38 INFO] (SDKD log:137) Call SweepOnce()
[1766,38 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1766,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[1781,73 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.15
[1781,73 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@111.25 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1781,74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1781,74 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.26 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,74 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1781,74 INFO] (SDKD log:137) "Options": {
[1781,75 INFO] (SDKD log:137) "DelayMin": 1,
[1781,75 INFO] (SDKD log:137) "TimeRes": 1,
[1781,75 INFO] (SDKD log:137) "IterWait": 1,
[1781,75 INFO] (SDKD log:137) "DelayMax": 10
[1781,76 INFO] (SDKD log:137) },
[1781,76 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1781,76 INFO] (SDKD log:137) "DS": {
[1781,76 INFO] (SDKD log:137) "KSize": 32,
[1781,77 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1781,77 INFO] (SDKD log:137) "VSize": 128,
[1781,78 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1781,78 INFO] (SDKD log:137) "Continuous": true,
[1781,78 INFO] (SDKD log:137) "Repeat": "REP",
[1781,78 INFO] (SDKD log:137) "Count": 15000
[1781,79 INFO] (SDKD log:137) }
[1781,79 INFO] (SDKD log:137) }
[1781,79 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,80 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1781,80 INFO] (SDKD log:137) Call SweepOnce()
[1781,80 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[1787,45 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.9
[1787,45 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@105.27 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1787,45 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1787,46 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.28 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1787,46 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1787,47 INFO] (SDKD log:137) "Options": {
[1787,47 INFO] (SDKD log:137) "DelayMin": 1,
[1787,47 INFO] (SDKD log:137) "TimeRes": 1,
[1787,47 INFO] (SDKD log:137) "IterWait": 1,
[1787,47 INFO] (SDKD log:137) "DelayMax": 10
[1787,48 INFO] (SDKD log:137) },
[1787,48 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1787,48 INFO] (SDKD log:137) "DS": {
[1787,49 INFO] (SDKD log:137) "KSize": 32,
[1787,49 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1787,49 INFO] (SDKD log:137) "VSize": 128,
[1787,50 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1787,50 INFO] (SDKD log:137) "Continuous": true,
[1787,50 INFO] (SDKD log:137) "Repeat": "REP",
[1787,50 INFO] (SDKD log:137) "Count": 15000
[1787,51 INFO] (SDKD log:137) }
[1787,51 INFO] (SDKD log:137) }
[1787,52 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1787,52 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1787,52 INFO] (SDKD log:137) Call SweepOnce()
[1787,52 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[1802,48 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.13
[1802,49 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@109.29 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1802,49 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1802,49 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.30 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1802,50 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1802,50 INFO] (SDKD log:137) "Options": {
[1802,50 INFO] (SDKD log:137) "DelayMin": 1,
[1802,50 INFO] (SDKD log:137) "TimeRes": 1,
[1802,51 INFO] (SDKD log:137) "IterWait": 1,
[1802,51 INFO] (SDKD log:137) "DelayMax": 10
[1802,51 INFO] (SDKD log:137) },
[1802,52 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1802,52 INFO] (SDKD log:137) "DS": {
[1802,52 INFO] (SDKD log:137) "KSize": 32,
[1802,52 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1802,53 INFO] (SDKD log:137) "VSize": 128,
[1802,53 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1802,53 INFO] (SDKD log:137) "Continuous": true,
[1802,53 INFO] (SDKD log:137) "Repeat": "REP",
[1802,54 INFO] (SDKD log:137) "Count": 15000
[1802,54 INFO] (SDKD log:137) }
[1802,55 INFO] (SDKD log:137) }
[1802,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1802,55 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1802,55 INFO] (SDKD log:137) Call SweepOnce()
[1802,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[1812,21 INFO] (SDKD log:137)
[1812,22 INFO] (SDKD log:137) Unhandle Exception: System.Security.Authentication.AuthenticationException: default
[1812,22 INFO] (SDKD log:137) at at Couchbase.IO.Strategies.DefaultIOStrategy.Authenticate(IConnection connection) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 78
[1812,22 INFO] (SDKD log:137) at Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 43
[1812,23 INFO] (SDKD log:137) at Couchbase.Core.Server.Send[T](IOperation`1 operation) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Server.cs:line 84
[1812,23 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 98
[1812,23 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 888
[1812,23 INFO] (SDKD log:137) at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\GetCommand.cs:line 18
[1812,24 INFO] (SDKD log:137) at Sdkd.Commands.KVCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
[1812,24 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) at c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 235
[1812,24 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.Run() at c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 156
[1812,24 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[1812,25 INFO] (SDKD log:137) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[1812,25 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[1812,26 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[1812,26 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart()
[1977,47 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: java.net.SocketException: Connection reset
        at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.Workload.makeHandle(Workload.java:91) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.KeyValueWorkload.makeHandleCommon(KeyValueWorkload.java:70) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.KeyValueWorkload.access$200(KeyValueWorkload.java:24) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.KeyValueWorkload$1$1.call(KeyValueWorkload.java:119) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_11]
        at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_11]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0_11]
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[na:1.8.0_11]
        at java.io.InputStreamReader.read(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedReader.fill(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.8.0_11]
        at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:152) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.sendMessage(Handle.java:173) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.negotiate(Handle.java:237) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.HandleFactory.createHandle(HandleFactory.java:43) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.Workload.makeHandle(Workload.java:89) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        ... 7 common frames omitted
[1977,55 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.31
[1977,55 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]



 Comments   
Comment by Jeff Morris [ 20/Aug/14 ]
This is likely caused by:
a) missing or deleted bucket - the server will return an authentication error if it cannot locate the bucket.
b) IO error that's currently being hidden by the AuthenticationException
c) Bucket name and/or credentials are invalid

There is nothing I can do for "a" - the server would have to change the message it returns and the test runner may be deleting buckets. For "b", i am working on a patch to handle this - the result may still be an uncaught exception though in SDKD, for "c" that's the domain of the caller to manage.

I am not sure this would classifiy as a "test blocker" - a test blocker should be a bug (and the same bug) that appears on every run of every test and not intermittently/situation-ally.
Comment by Andrei Baranouski [ 20/Aug/14 ]
seems like a) and b) are not my cases. the test is performed without any problems for a while, and then fails(bucket is not removed)
it's reproduced every time with this scenario. will try some other.
Comment by Jeff Morris [ 20/Aug/14 ]
Andrei -

Can you describe in words this particular test (I am not a sdkd expert :))? Does this specific test fail every time (a good thing actually)?

My patch when completed, should fix the issue where the client tries to auth against a node that has been removed from the cluster - this causes most of the Auth exceptions.

Thanks,

Jeff
Comment by Jeff Morris [ 20/Aug/14 ]
This where the code failed:
2014-08-20 10:32:18,479 [21] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authentication for socket fef9f34e-6c36-42a3-8970-24bd5487a28c failed: Timed out.
2014-08-20 10:32:18,479 [21] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Could not authenticate default using Couchbase.Authentication.SASL.CramMd5Mechanism - d8041012-7830-45dc-a97c-2b64a0a3215b.

It "Timed out", which means the remote host failed to respond within the grace period. Likely the server has been ejected from the cluster at that point.
Comment by Jeff Morris [ 20/Aug/14 ]
http://review.couchbase.org/#/c/40772/
Comment by Andrei Baranouski [ 21/Aug/14 ]
fixed




[NCBC-591] NullReferenceException in Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetBucketConfig Created: 19/Aug/14  Updated: 21/Aug/14  Resolved: 19/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File config-with-1-node.json     File config-with-2-nodes.json     Zip Archive NCBC-591.log.zip    

 Description   
$ git log
commit 07c4c72cc7a34cc46149cf9be357c84913113fae
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 18 22:31:37 2014 -0700

    NCBC-589: ThrowInvalidOperationException in CouchbaseConfigContext.LoadConfig during rebalance

    Change-Id: I813092f07bb6fde414ea4e05a67239173a540568

commit 8ba11bbadd0d9d2b3621840b531e350a6b506760
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 18 22:15:49 2014 -0700

    Check that length is positive in TypeSerializer.Deserialize

    Change-Id: Ic229fbdd0362593ac7577350cc04f549380ac7f5

commit 5c2c0e3589b5f0099fc8048de8b398c28f6eb04c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Tue Aug 12 16:48:03 2014 -0700

    NCBC-581: System.InvalidOperationException when server list is empty

    Change-Id: I8fdef46b4057bc9f3b898b789fcccd78f689fdb6
    Reviewed-on: http://review.couchbase.org/40546




I changed the sdkd-net code to a primitive version, and now each handler(read as thread of cb commands) I use separate CouchbaseCluster and CouchbaseBucket for each handler

using (var client = new CouchbaseCluster(config)){
   using (var SdkHandle = (CouchbaseBucket)client.OpenBucket("default", ""))
        {
            resp = new Sdkd.Protocol.Response(req, 0);
            SendMessage(resp);

            while (true)
            {
                req = GetMessage(true);
                if (req == null)
                {
                    HLogger.Warn("Socket closed without sending CLOSEHANDLE");
                    ReleaseHandle(SdkHandle);
                    return;
                }

                HLogger.Warn("Got command {0}", req.Command);
                HLogger.Warn("Command data: {0}", req.CommandData);

                try
                {
                    if (DispatchCommand(req, SdkHandle) == false)
                    {
                        ReleaseHandle(SdkHandle);
                        HLogger.Warn("DispatchCommand(req, SdkHandle) == false");
                        return;
                    }
                }
                catch (SdkdProtocolException exc)
                {
                    SendMessage(new Response(req, exc.ProtoErr));
                }
            }
        }


( I still face with http://www.couchbase.com/issues/browse/NCBC-575 problem when I use the same CouchbaseCluster for different CouchbaseBucket instances. will add my thoughts there)

c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,12 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/count=2
    workload=HybridWorkloadGroup
    rebalance/mode=out
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-19-14\067477\8af0690bf5e9f06fa0c37ccbebf0379d

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-count 2 --workload HybridWorkloadGroup
 \ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1,34 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1,65 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,66 INFO] (RunContext run:124) Starting cluster and driver
[1,66 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,66 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1,66 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[1,82 INFO] (SDKD log:137) --- Logging Self-Test ---
[1,82 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[1,82 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[1,82 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[1,83 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[1,83 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,19 INFO] (SDKD log:137) Call SweepOnce()
[2,19 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,19 INFO] (SDKD log:137) Call SweepOnce()
[2,20 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,20 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[2,36 INFO] (SDKD log:137) Call SweepOnce()
[2,37 INFO] (SDKD log:137) Call SweepOnce()
[2,37 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[7,66 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.145:8091,ns_1@10.3.4.145>
[8,10 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.146:8091,ns_1@10.3.4.146>
[8,52 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[8,99 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[8,99 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[8,99 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[9,00 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,93 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[10,94 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[10,94 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[10,94 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[10,94 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[10,94 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[10,95 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[10,95 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[11,65 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@325c364d
[11,65 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@447b4664
[11,66 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@2011af89
[11,66 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@2d0c6b05
[11,66 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@167a53aa
[26,07 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[34,03 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[44,85 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[50,40 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[51,26 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[51,26 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[51,47 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[51,69 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[60,67 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[63,17 INFO] (RunContext run:143) Driver and cluster initialized
[63,38 INFO] (RunContext call:167) Running scenario..
[63,38 INFO] (Scenario run:72) Starting RAMP phase
[63,38 INFO] (Workload setupDesign:63) Creating design test_design
[64,51 INFO] (Workload setupDesign:80) Design creation done
[64,51 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[64,51 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[64,51 INFO] (SDKD log:137) Call SweepOnce()
[64,52 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[64,52 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[64,54 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[64,54 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[64,57 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[64,57 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[71,93 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[71,93 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {Options={DelayMin=0, TimeRes=0, IterWait=1, DelayMax=0}, Schema={InflateLevel=40, InflateContent=meh}, DSType=DSTYPE_SEEDED, DS={KSize=12, KSeed=ViewFillerSeed, VSize=512, VSeed=Vie
wFillerSeed, Continuous=false, Repeat=rep, Count=1000}}
[71,93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[71,94 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[71,94 INFO] (SDKD log:137) "Options": {
[71,94 INFO] (SDKD log:137) "DelayMin": 0,
[71,94 INFO] (SDKD log:137) "TimeRes": 0,
[71,95 INFO] (SDKD log:137) "IterWait": 1,
[71,95 INFO] (SDKD log:137) "DelayMax": 0
[71,95 INFO] (SDKD log:137) },
[71,95 INFO] (SDKD log:137) "Schema": {
[71,96 INFO] (SDKD log:137) "InflateLevel": 40,
[71,96 INFO] (SDKD log:137) "InflateContent": "meh"
[71,96 INFO] (SDKD log:137) },
[71,97 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[71,97 INFO] (SDKD log:137) "DS": {
[71,97 INFO] (SDKD log:137) "KSize": 12,
[71,97 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[71,98 INFO] (SDKD log:137) "VSize": 512,
[71,98 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[71,99 INFO] (SDKD log:137) "Continuous": false,
[71,99 INFO] (SDKD log:137) "Repeat": "rep",
[72,00 INFO] (SDKD log:137) "Count": 1000
[72,00 INFO] (SDKD log:137) }
[72,00 INFO] (SDKD log:137) }
[256,45 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[256,46 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[256,46 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[256,46 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[256,47 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[256,47 INFO] (Workload call:112) Preloading items into cluster before workload
[256,47 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[256,47 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[256,48 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[256,48 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[256,48 INFO] (SDKD log:137) Call SweepOnce()
[256,49 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[256,49 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[256,49 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[265,65 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[265,65 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@102.5 => {Options={DelayMin=0, TimeRes=0, IterWait=100, DelayMax=0}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=false, Repeat=REP, Coun
t=15000}}
[265,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[265,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[265,67 INFO] (SDKD log:137) "Options": {
[265,67 INFO] (SDKD log:137) "DelayMin": 0,
[265,67 INFO] (SDKD log:137) "TimeRes": 0,
[265,67 INFO] (SDKD log:137) "IterWait": 100,
[265,67 INFO] (SDKD log:137) "DelayMax": 0
[265,67 INFO] (SDKD log:137) },
[265,68 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[265,68 INFO] (SDKD log:137) "DS": {
[265,68 INFO] (SDKD log:137) "KSize": 32,
[265,68 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[265,69 INFO] (SDKD log:137) "VSize": 128,
[265,69 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[265,69 INFO] (SDKD log:137) "Continuous": false,
[265,69 INFO] (SDKD log:137) "Repeat": "REP",
[265,69 INFO] (SDKD log:137) "Count": 15000
[265,70 INFO] (SDKD log:137) }
[265,70 INFO] (SDKD log:137) }
[405,30 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[1650,02 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@102.5 => {[OK]: 14985, [MEMD:GENERIC]: 15}
[1650,02 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[1650,02 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[1650,03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[1650,03 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1744713491
[1650,03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.8 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.11 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.12 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,04 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.10 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,05 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.9 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,05 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1744713491
[1650,06 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.16 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,06 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.13 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,06 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.15 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,06 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.14 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1650,07 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[1650,07 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,07 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,08 INFO] (SDKD log:137) Call SweepOnce()
[1650,08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,08 INFO] (SDKD log:137) Call SweepOnce()
[1650,08 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,09 INFO] (SDKD log:137) Call SweepOnce()
[1650,09 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,09 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,10 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[1650,10 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[1650,10 INFO] (SDKD log:137) Call SweepOnce()
[1650,11 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,11 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,11 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[1650,12 INFO] (SDKD log:137) Call SweepOnce()
[1650,12 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,13 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,13 INFO] (SDKD log:137) Call SweepOnce()
[1650,13 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,13 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,14 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[1650,14 INFO] (SDKD log:137) Call SweepOnce()
[1650,14 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,15 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,15 INFO] (SDKD log:137) Call SweepOnce()
[1650,15 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,15 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,16 INFO] (SDKD log:137) Call SweepOnce()
[1650,16 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,16 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1650,17 INFO] (SDKD log:137) Call SweepOnce()
[1650,17 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1650,17 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[1650,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[1650,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[1650,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[1650,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[1650,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[1655,77 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[1655,78 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@103.17 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1655,78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1655,78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1655,78 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.18 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1655,78 INFO] (SDKD log:137) "Options": {
[1655,79 INFO] (SDKD log:137) "DelayMin": 1,
[1655,79 INFO] (SDKD log:137) "TimeRes": 1,
[1655,79 INFO] (SDKD log:137) "IterWait": 1,
[1655,79 INFO] (SDKD log:137) "DelayMax": 10
[1655,79 INFO] (SDKD log:137) },
[1655,79 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1655,79 INFO] (SDKD log:137) "DS": {
[1655,79 INFO] (SDKD log:137) "KSize": 32,
[1655,80 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1655,80 INFO] (SDKD log:137) "VSize": 128,
[1655,80 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1655,80 INFO] (SDKD log:137) "Continuous": true,
[1655,80 INFO] (SDKD log:137) "Repeat": "REP",
[1655,80 INFO] (SDKD log:137) "Count": 15000
[1655,80 INFO] (SDKD log:137) }
[1655,81 INFO] (SDKD log:137) }
[1655,81 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1655,81 INFO] (SDKD log:137) Call SweepOnce()
[1655,81 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1655,81 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[1661,66 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.11
[1661,66 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@107.19 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1661,67 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1661,67 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.20 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1661,67 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1661,68 INFO] (SDKD log:137) "Options": {
[1661,68 INFO] (SDKD log:137) "DelayMin": 1,
[1661,68 INFO] (SDKD log:137) "TimeRes": 1,
[1661,68 INFO] (SDKD log:137) "IterWait": 1,
[1661,69 INFO] (SDKD log:137) "DelayMax": 10
[1661,69 INFO] (SDKD log:137) },
[1661,69 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1661,69 INFO] (SDKD log:137) "DS": {
[1661,70 INFO] (SDKD log:137) "KSize": 32,
[1661,70 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1661,70 INFO] (SDKD log:137) "VSize": 128,
[1661,70 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1661,70 INFO] (SDKD log:137) "Continuous": true,
[1661,71 INFO] (SDKD log:137) "Repeat": "REP",
[1661,71 INFO] (SDKD log:137) "Count": 15000
[1661,71 INFO] (SDKD log:137) }
[1661,71 INFO] (SDKD log:137) }
[1661,72 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1661,72 INFO] (SDKD log:137) Call SweepOnce()
[1661,72 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1661,72 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[1667,55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.8
[1667,55 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@104.21 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1667,55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1667,55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1667,56 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.22 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1667,56 INFO] (SDKD log:137) "Options": {
[1667,56 INFO] (SDKD log:137) "DelayMin": 1,
[1667,56 INFO] (SDKD log:137) "TimeRes": 1,
[1667,57 INFO] (SDKD log:137) "IterWait": 1,
[1667,57 INFO] (SDKD log:137) "DelayMax": 10
[1667,57 INFO] (SDKD log:137) },
[1667,57 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1667,57 INFO] (SDKD log:137) "DS": {
[1667,58 INFO] (SDKD log:137) "KSize": 32,
[1667,58 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1667,58 INFO] (SDKD log:137) "VSize": 128,
[1667,58 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1667,59 INFO] (SDKD log:137) "Continuous": true,
[1667,59 INFO] (SDKD log:137) "Repeat": "REP",
[1667,59 INFO] (SDKD log:137) "Count": 15000
[1667,59 INFO] (SDKD log:137) }
[1667,60 INFO] (SDKD log:137) }
[1667,60 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1667,61 INFO] (SDKD log:137) Call SweepOnce()
[1667,61 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1667,61 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[1672,50 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.12
[1672,51 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.23 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1672,51 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1672,51 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1672,52 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.24 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1672,52 INFO] (SDKD log:137) "Options": {
[1672,53 INFO] (SDKD log:137) "DelayMin": 1,
[1672,53 INFO] (SDKD log:137) "TimeRes": 1,
[1672,53 INFO] (SDKD log:137) "IterWait": 1,
[1672,53 INFO] (SDKD log:137) "DelayMax": 10
[1672,53 INFO] (SDKD log:137) },
[1672,53 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1672,54 INFO] (SDKD log:137) "DS": {
[1672,54 INFO] (SDKD log:137) "KSize": 32,
[1672,54 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1672,54 INFO] (SDKD log:137) "VSize": 128,
[1672,55 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1672,55 INFO] (SDKD log:137) "Continuous": true,
[1672,55 INFO] (SDKD log:137) "Repeat": "REP",
[1672,55 INFO] (SDKD log:137) "Count": 15000
[1672,56 INFO] (SDKD log:137) }
[1672,56 INFO] (SDKD log:137) }
[1672,56 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1672,56 INFO] (SDKD log:137) Call SweepOnce()
[1672,56 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1672,56 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[1687,65 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.10
[1687,65 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@106.25 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1687,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1687,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1687,66 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.26 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1687,66 INFO] (SDKD log:137) "Options": {
[1687,67 INFO] (SDKD log:137) "DelayMin": 1,
[1687,67 INFO] (SDKD log:137) "TimeRes": 1,
[1687,67 INFO] (SDKD log:137) "IterWait": 1,
[1687,67 INFO] (SDKD log:137) "DelayMax": 10
[1687,68 INFO] (SDKD log:137) },
[1687,68 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1687,68 INFO] (SDKD log:137) "DS": {
[1687,68 INFO] (SDKD log:137) "KSize": 32,
[1687,69 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1687,69 INFO] (SDKD log:137) "VSize": 128,
[1687,69 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1687,69 INFO] (SDKD log:137) "Continuous": true,
[1687,69 INFO] (SDKD log:137) "Repeat": "REP",
[1687,70 INFO] (SDKD log:137) "Count": 15000
[1687,70 INFO] (SDKD log:137) }
[1687,70 INFO] (SDKD log:137) }
[1687,70 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1687,70 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1687,71 INFO] (SDKD log:137) Call SweepOnce()
[1687,71 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[1702,13 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.9
[1702,13 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@105.27 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1702,14 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1702,14 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1702,14 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.28 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1702,14 INFO] (SDKD log:137) "Options": {
[1702,14 INFO] (SDKD log:137) "DelayMin": 1,
[1702,15 INFO] (SDKD log:137) "TimeRes": 1,
[1702,15 INFO] (SDKD log:137) "IterWait": 1,
[1702,15 INFO] (SDKD log:137) "DelayMax": 10
[1702,15 INFO] (SDKD log:137) },
[1702,16 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1702,16 INFO] (SDKD log:137) "DS": {
[1702,16 INFO] (SDKD log:137) "KSize": 32,
[1702,16 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1702,16 INFO] (SDKD log:137) "VSize": 128,
[1702,17 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1702,17 INFO] (SDKD log:137) "Continuous": true,
[1702,17 INFO] (SDKD log:137) "Repeat": "REP",
[1702,17 INFO] (SDKD log:137) "Count": 15000
[1702,18 INFO] (SDKD log:137) }
[1702,18 INFO] (SDKD log:137) }
[1702,18 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1702,18 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1702,18 INFO] (SDKD log:137) Call SweepOnce()
[1702,19 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[1808,14 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.13
[1808,15 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@109.29 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1808,15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1808,15 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.30 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1808,15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1808,16 INFO] (SDKD log:137) "Options": {
[1808,16 INFO] (SDKD log:137) "DelayMin": 1,
[1808,16 INFO] (SDKD log:137) "TimeRes": 1,
[1808,17 INFO] (SDKD log:137) "IterWait": 1,
[1808,17 INFO] (SDKD log:137) "DelayMax": 10
[1808,17 INFO] (SDKD log:137) },
[1808,17 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1808,18 INFO] (SDKD log:137) "DS": {
[1808,18 INFO] (SDKD log:137) "KSize": 32,
[1808,19 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1808,19 INFO] (SDKD log:137) "VSize": 128,
[1808,19 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1808,19 INFO] (SDKD log:137) "Continuous": true,
[1808,20 INFO] (SDKD log:137) "Repeat": "REP",
[1808,20 INFO] (SDKD log:137) "Count": 15000
[1808,20 INFO] (SDKD log:137) }
[1808,20 INFO] (SDKD log:137) }
[1808,21 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1808,21 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1808,22 INFO] (SDKD log:137) Call SweepOnce()
[1808,22 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[2014,15 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.16
[2014,15 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@112.31 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[2014,15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[2014,16 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[2014,16 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.32 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[2014,16 INFO] (SDKD log:137) "Options": {
[2014,16 INFO] (SDKD log:137) "DelayMin": 1,
[2014,16 INFO] (SDKD log:137) "TimeRes": 1,
[2014,16 INFO] (SDKD log:137) "IterWait": 1,
[2014,17 INFO] (SDKD log:137) "DelayMax": 10
[2014,17 INFO] (SDKD log:137) },
[2014,17 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[2014,17 INFO] (SDKD log:137) "DS": {
[2014,17 INFO] (SDKD log:137) "KSize": 32,
[2014,18 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[2014,18 INFO] (SDKD log:137) "VSize": 128,
[2014,18 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[2014,18 INFO] (SDKD log:137) "Continuous": true,
[2014,18 INFO] (SDKD log:137) "Repeat": "REP",
[2014,19 INFO] (SDKD log:137) "Count": 15000
[2014,19 INFO] (SDKD log:137) }
[2014,19 INFO] (SDKD log:137) }
[2014,19 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2014,19 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[2014,19 INFO] (SDKD log:137) Call SweepOnce()
[2014,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120
[2115,71 INFO] (SDKD log:137)
[2115,72 INFO] (SDKD log:137) Unhandle Exception: System.NullReferenceException: Object reference not set to an instance of the object.
[2115,72 INFO] (SDKD log:137) в Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetBucketConfig(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Provid
ers\Streaming\HttpStreamingProvider.cs:line 247
[2115,72 INFO] (SDKD log:137) at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetConfig(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\St
reaming\HttpStreamingProvider.cs:line 54
[2115,73 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 159
[2115,73 INFO] (SDKD log:137) в Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 84
[2115,73 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 134
[2115,73 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[2115,73 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[2115,74 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[2115,74 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[2115,74 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart()
[2151,61 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_11]
        at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_11]
[2151,62 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.33
[2151,63 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]



 Comments   
Comment by Jeff Morris [ 19/Aug/14 ]
>>I changed the sdkd-net code to a primitive version, and now each handler(read as thread of cb commands) I use separate CouchbaseCluster and CouchbaseBucket for each handler

Good, this should make it easier to isolate issues. Let me dig through the logs and see what I find.
Comment by Jeff Morris [ 19/Aug/14 ]
The issue appears to be that when a cluster is a single node, it returns $HOST as the hostname in the nodes element of the config. If the cluster is greater than 1, then it returns the proper hostname for each node. The resolution for $HOST fails, thus making the BucketConfig object null when it's referenced. Please see attached JSON files.

Note this is using CB 3.0 beta 2.
Comment by Matt Ingenthron [ 19/Aug/14 ]
The $HOST is expected. You might recall that from a while ago. The client library is supposed to swap in the IP/hostname it used to retrieve that config.
Comment by Jeff Morris [ 19/Aug/14 ]
Yeah, I remember $HOST...$HOST is interesting, because it may or may not be in the config: after adding the second node, I removed it making it a single node cluster and no $HOST was present - it was using the IP again.
Comment by Jeff Morris [ 19/Aug/14 ]
http://review.couchbase.org/#/c/40742/




[NCBC-577] SDK 1.3.8 doesn't support uuid with bucket name in View query against 3.0 Created: 07/Aug/14  Updated: 21/Aug/14  Resolved: 12/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 1.3.6
Fix Version/s: 1.3.8

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Microsoft Word encoding-issues-dotnet-sdk-137-couchabase-server-3.docx     PNG File jmorris-vs2012.png     Zip Archive system.uri-diff-output-in-console-vs-library-assemblies.zip     PNG File uri-mystery.png    

 Description   
run https://github.com/couchbase/couchbase-net-client/commit/86fd1c588df15611b557bfa7e401581e005c188e
against 3.0.0-1105

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/2/console

[176.39 INFO] (SDKD log:137) [Couchbase.DefaultHttpClient|Warn] System.Net.WebException: The remote server returned an error: (404) Not Found.
[176.39 INFO] (SDKD log:137) +++ Received exception of ID 0
[176.39 INFO] (SDKD log:137) [Couchbase.CouchbaseViewHandler|Warn] Design document not found, body: {"error":"not_found","reason":"missing"}
[176.39 INFO] (SDKD log:137)  


Jeff, as I understand 1.3.8 should support 3.0 CB as well?

then, the same fix for https://www.couchbase.com/issues/browse/NCBC-545 ( https://github.com/couchbase/couchbase-net-client/commit/e5ead27c1efa5d33bb4d0ba9f55d424ffd9d3c79) should be adjusted for 1.3.8?



 Comments   
Comment by Jeff Morris [ 07/Aug/14 ]
Yes, this is a bug specific to 3.0.
Comment by Jeff Morris [ 08/Aug/14 ]
Andrei -

From my unit tests this appears to not be a client error. The cluster you are running the tests against is missing the design doc you are querying: "Design document not found, body: {"error":"not_found","reason":"missing"}"

What I would do:

a) Check that the cluster indeed has the design doc and views sdkqe-net is using
b) Make sure it's not an issue with dev vs production: publish the view if it is

Also, including the url that has failed really helps debugging if you can locate it (along with the view name).

Assigning back to you.

Thanks,

Jeff

Comment by Andrei Baranouski [ 09/Aug/14 ]
Hi Jeff,

src/Couchbase/DefaultHttpClient.cs

I print out uri requests for HttpWebRequest

@@ -198,10 +198,11 @@ namespace Couchbase
                 queryString += string.Join("&", select);
 #endif
                 lock (client)
                 {
                     var uri = new Uri(client.BaseAddress + "/" + path + queryString);
+Console.WriteLine(uri);
                     var request = client.GetWebRequest(uri, client.BaseAddress.GetHashCode().ToString()) as HttpWebRequest;
                     request.Accept = "application/json";
                     request.ContentType = "application/json; charset=utf-8";
                     return request;
                 }


46.97 INFO] (SDKD log:137) http://10.3.121.136:8092/default+0218b57a1309c1d9a9bdf94f90057d27/_design/test_design/_view/test_view?
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/17/console


http://10.3.121.136:8092/default+0218b57a1309c1d9a9bdf94f90057d27/_design/test_design/_view/test_view - with such URI I get {"error":"not_found","reason":"missing"}
http://10.3.121.136:8092/default%2B0218b57a1309c1d9a9bdf94f90057d27/_design/test_design/_view/test_view it's a valid URI


"couchApiBase":"http://10.3.3.206:8092/default%2B0218b57a1309c1d9a9bdf94f90057d27"
Comment by Jeff Morris [ 09/Aug/14 ]
Andrei -

Thanks for the details, i'll investigate further!

-Jeff
Comment by Jeff Morris [ 09/Aug/14 ]
This is the output I get when I write the url to stdout: VIEW QUERY: http://192.168.56.101:8092/default%2Beac6cf9b998068ca424c5b48baaee12a/_design/cities/?

Are you sure your not generating that url (with the "+") from sdkqe-net? Could also be perhaps some environment difference?
Comment by Andrei Baranouski [ 09/Aug/14 ]
added logs in CouchbasePool.cs
        protected IMemcachedNode CreateNode(IPEndPoint endpoint, ISaslAuthenticationProvider auth, Dictionary<string, object> nodeInfo)
        {
            log.WarnFormat("Creating node {0}", this);
            string couchApiBase;
            if (!nodeInfo.TryGetValue("couchApiBase", out couchApiBase) || String.IsNullOrEmpty(couchApiBase))
            {
                return new CouchbaseNode(endpoint, configuration.SocketPool, auth);
            }
            Console.WriteLine("in CreateNode" + couchApiBase);
            Console.WriteLine("in CreateNode" + new Uri(couchApiBase));
            return new CouchbaseNode(endpoint, new Uri(couchApiBase), this.configuration, auth);
        }

[37.33 INFO] (SDKD log:137) in CreateNodehttp://10.3.121.134:8092/default%2Bd2da2be3dfa338b5bea90d3f63a40362
[37.33 INFO] (SDKD log:137) in CreateNodehttp://10.3.121.134:8092/default+d2da2be3dfa338b5bea90d3f63a40362

it means that new Uri() encode "%2B" in "+"

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/22/console

Comment by Jeff Morris [ 09/Aug/14 ]
My output:

in CreateNodehttp://192.168.56.101:8092/default%2Beac6cf9b998068ca424c5b48baaee12a
in CreateNodehttp://192.168.56.102:8092/default%2Beac6cf9b998068ca424c5b48baaee12a

The problem is your probably using an older version of the server and this was fixed at the server level!

My cluster is using:

Version: 3.0.0 Enterprise Edition (build-1098)
Cluster State ID: 048-041-39

I'll try to find the MB. We should probably testing against the latest server builds for 3.0, since a lot of this stuff is probably fixed.

Comment by Andrei Baranouski [ 09/Aug/14 ]
it's strange

in run http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/17/consoleFull I used 3.0.0-1105-rel (Beta2 build)
I install CB directly in jenkins run
c:\jenkins\workspace\.net-sdkdclient-ng-andrei\testrunner>python.exe scripts/install.py -i net-sdkdclient-ng-andrei.ini -p version=3.0.0-1105-rel,product=cb,parallel=True,vbuckets=1024

and in this version I see :
[46.97 INFO] (SDKD log:137) http://10.3.3.206:8092/default+0218b57a1309c1d9a9bdf94f90057d27/_design/test_design/_view/test_view?
Comment by Jeff Morris [ 09/Aug/14 ]
Andrei - try this: http://review.couchbase.org/#/c/40485/1

This should be safe and sane fix.
Comment by Andrei Baranouski [ 10/Aug/14 ]
Jeff, it doesn't work because
couchApiBase contains "%2B" but not "+"
couchApiBase = couchApiBase.Replace("+", "%2B"); ie it does not affect the result.

believe that the solution is simple but not obvious)
Comment by Jeff Morris [ 10/Aug/14 ]
The mystery continues: depending upon the version of the .NET Framework you compile against, you get different output from Uri.ToString().

Note that the ".NET Version" output to console isn't indicating which .NET version, but really just the CLR version (my code sucks).

Even stranger, according to this test, when I run the Couchbase.Tests project (it's set to use .NET Framework 4.0), I should be getting the url encoded with "+" - but I don't (lines 325 and 326 of CouchbasePool.cs):

Console.WriteLine("in CreateNode 1: " + couchApiBase);
Console.WriteLine("in CreateNode 2: " + new Uri(couchApiBase));

[log4net.Repository.Hierarchy.Hierarchy]
in CreateNode 1: http://192.168.56.101:8092/default%2Beac6cf9b998068ca424c5b48baaee12a
in CreateNode 2: http://192.168.56.101:8092/default%2Beac6cf9b998068ca424c5b48baaee12a

The best solution is probably to check and convert the url string going out. I'll upload a patch.
Comment by Jeff Morris [ 10/Aug/14 ]
It appears that the System.Uri behaves differently in different types of assemblies. Attached is a zipped solution with two projects: a) a console application w/a main() method and b) client library w/unit test. Each runs roughly the same code, however how System.Uri encodes urls is different.

In the console project:
CLR Version: 4.0.30319.34014
.NET Version: 4.0.0.0
http://10.3.121.134:8092/default%2Bd2da2be3dfa338b5bea90d3f63a40362
http://10.3.121.134:8092/default+d2da2be3dfa338b5bea90d3f63a40362

If you execute the unit test:
CLR Version: 4.0.30319.34014
.NET Version: 4.0.0.0
http://10.3.121.134:8092/default%2Bd2da2be3dfa338b5bea90d3f63a40362
http://10.3.121.134:8092/default%2Bd2da2be3dfa338b5bea90d3f63a40362

This is why the unit tests run correctly, but the situational tests fail (SDKQE-NET is a console app).


Comment by Jeff Morris [ 11/Aug/14 ]
Hi Andrei -

Try this: http://review.couchbase.org/#/c/40511/

It should do the trick.

Thanks,

Jeff
Comment by Andrei Baranouski [ 12/Aug/14 ]
Hi Jeff,

URI is still built like

[357.18 INFO] (SDKD log:137) http://10.3.3.206:8092/default+4aa397ed239d973a8ccac45320abd80c/_design/test_design/_view/test_view?
[357.18 INFO] (SDKD log:137) http://10.3.3.206:8092/default+4aa397ed239d973a8ccac45320abd80c/_design/test_design/_view/test_view?limit=1

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/69/console

to build couchbase-net-client & sdkd-net I use the following settings
set MSBUILD_PATH=C:\Program Files (x86)\MSBuild\12.0\Bin\amd64\;C:\Windows\Microsoft.NET\Framework64\v4.0.30319"
SET PATH=%MSBUILD_PATH%;%PATH%

IP: 10.3.121.13
Comment by Jeff Morris [ 12/Aug/14 ]
Looking deeper into this, I may not be able to fix it easily - in System.Net.WebClient.cs (from decompiled .NET source):

public string BaseAddress
    {
      get
      {
        if (!(this.m_baseAddress == (Uri) null))
          return this.m_baseAddress.ToString();
        else
          return string.Empty;
      }
      set
      {
        if (value != null)
        {
          if (value.Length != 0)
          {
            try
            {
              this.m_baseAddress = new Uri(value);
              return;
            }
            catch (UriFormatException ex)
            {
              throw new ArgumentException(SR.GetString("net_webclient_invalid_baseaddress"), "value", (Exception) ex);
            }
          }
        }
        this.m_baseAddress = (Uri) null;
      }
    }

When you set the BaseAddress string property, it casts it to a Uri (which contains the bug) and calls it's ToString method. It's kinda sad MS didn't/doesn't expose the m_baseAddress field.

Comment by Jeff Morris [ 12/Aug/14 ]
It looks like I would have to rewrite nearly all of the System.Net namespace. I think it might be better to handle this via documentation and support: the .NET 1.X SDK only supports Couchbase Server 3.0 from .NET Framework 4.5 onward.

Comment by Jeff Morris [ 12/Aug/14 ]
Or another approach would be to simply remove the uuid and %2B characters. Does the server require it now?
Comment by Jeff Morris [ 12/Aug/14 ]
Andrei -

Try putting this configuration element into SDK-nets App.Config file:

<uri>
    <iriParsing enabled="true"/>
</uri>

After you have done this, run the tests again. If this works, I'll try a programmatic solution (if it exists).

Thanks,

Jeff
Comment by Jeff Morris [ 12/Aug/14 ]
New patchset: git fetch ssh://jmorris@review.couchbase.org:29418/couchbase-net-client refs/changes/11/40511/5 && git checkout FETCH_HEAD

I tested locally on clients using .NET 4, 4.5 and 4.5.1. This is the "programmatic" implementation mentioned in previous comment.
Comment by Andrei Baranouski [ 13/Aug/14 ]
awesome, it works! http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng-andrei/71/




[NCBC-588] System.InvalidOperationException: Sequence contains no matching element in Couchbase.Configuration.ConfigContextBase.GetServer() after rebalance Created: 16/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-588.zip    
Issue Links:
Gantt: finish-start

 Description   
I took a few commits ahead from gerrit


commit 2c00a9d3a397f5c1eddc56238cf4a19e6e6618dd
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 11 18:41:08 2014 -0700

    Add config files to tests

    Change-Id: Ie093eac1da4ffc742796ffd4df8a9c63cc1c9c2f

commit 5b18623a7c13fb16f21476fe77eaf9a192e92eb2
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 11 17:20:19 2014 -0700

    Complete IBucket implementation for Memcached and Couchbase Buckets

    Change-Id: If6fb47b9e4a5201e625f155dadf45696b8666674

commit a6001260b0437a9779f7b7730e857b77ac9aea31
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Aug 6 15:31:06 2014 -0700

    NCBC-542: Provide Observe method on IBucket

    Change-Id: Iecfbc02490b9ce237d6fbe16619e5aa252566bd7

commit 321940886cffcdc30225201ea243d8c6e4df6da4
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Aug 6 12:31:12 2014 -0700

    NCBC-576: Do not retry operation when unknown status is returned.

    Change-Id: Ic9afeb1373d9e7804cad3c2df0521b162956326d
    Reviewed-on: http://review.couchbase.org/40347
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Matt Ingenthron <matt@couchbase.com>


c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0.31 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-16-14\052918\efcd0dc3177c5d7533647e7a3fe8ee7a

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1.54 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1.97 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1.99 INFO] (RunContext run:124) Starting cluster and driver
[2.00 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[2.00 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[2.00 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2.31 INFO] (SDKD log:137) --- Logging Self-Test ---
[2.33 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[2.34 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[2.34 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[2.34 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[2.35 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2.53 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2.55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2.83 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.17929"}}
[5.06 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[5.16 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[5.17 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[5.17 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[5.17 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[5.75 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[5.76 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[5.77 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[5.79 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[5.79 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[5.80 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[5.80 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[5.80 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[6.01 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@6b96bac4
[6.02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@3dde8aa8
[6.02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@43a4181c
[6.02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@56b61c3
[6.04 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@1de4eb5b
[17.65 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[28.31 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[35.77 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[51.97 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[52.01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[53.07 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[53.08 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[53.09 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[53.11 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[61.02 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[61.37 INFO] (RunContext run:143) Driver and cluster initialized
[61.40 INFO] (RunContext call:167) Running scenario..
[61.41 INFO] (Scenario run:72) Starting RAMP phase
[61.41 INFO] (Workload setupDesign:63) Creating design test_design
[61.74 INFO] (Workload setupDesign:80) Design creation done
[61.74 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[61.75 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[61.76 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[61.77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[61.79 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[61.79 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[61.82 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[61.82 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[62.22 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[62.23 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12
}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[62.24 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[62.24 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[62.24 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[62.25 INFO] (SDKD log:137) "DS": {
[62.25 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[62.25 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[62.26 INFO] (SDKD log:137) "Count": 1000,
[62.26 INFO] (SDKD log:137) "Continuous": false,
[62.26 INFO] (SDKD log:137) "VSize": 512,
[62.26 INFO] (SDKD log:137) "Repeat": "rep",
[62.27 INFO] (SDKD log:137) "KSize": 12
[62.27 INFO] (SDKD log:137) },
[62.27 INFO] (SDKD log:137) "Schema": {
[62.27 INFO] (SDKD log:137) "InflateContent": "meh",
[62.28 INFO] (SDKD log:137) "InflateLevel": 40
[62.28 INFO] (SDKD log:137) },
[62.28 INFO] (SDKD log:137) "Options": {
[62.28 INFO] (SDKD log:137) "DelayMax": 0,
[62.29 INFO] (SDKD log:137) "DelayMin": 0,
[62.29 INFO] (SDKD log:137) "IterWait": 1,
[62.29 INFO] (SDKD log:137) "TimeRes": 0
[62.29 INFO] (SDKD log:137) }
[62.30 INFO] (SDKD log:137) }
[64.44 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[64.44 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[64.45 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[64.45 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[64.46 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[64.46 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[64.46 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[64.46 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[64.47 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[64.47 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[64.47 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[64.48 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[64.49 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[64.49 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@102.5 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12
}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[64.50 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[64.50 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[64.51 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[64.51 INFO] (SDKD log:137) "DS": {
[64.51 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[64.52 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[64.52 INFO] (SDKD log:137) "Count": 1000,
[64.52 INFO] (SDKD log:137) "Continuous": false,
[64.52 INFO] (SDKD log:137) "VSize": 512,
[64.53 INFO] (SDKD log:137) "Repeat": "rep",
[64.53 INFO] (SDKD log:137) "KSize": 12
[64.53 INFO] (SDKD log:137) },
[64.54 INFO] (SDKD log:137) "Schema": {
[64.54 INFO] (SDKD log:137) "InflateContent": "meh",
[64.54 INFO] (SDKD log:137) "InflateLevel": 40
[64.54 INFO] (SDKD log:137) },
[64.55 INFO] (SDKD log:137) "Options": {
[64.55 INFO] (SDKD log:137) "DelayMax": 0,
[64.55 INFO] (SDKD log:137) "DelayMin": 0,
[64.56 INFO] (SDKD log:137) "IterWait": 1,
[64.56 INFO] (SDKD log:137) "TimeRes": 0
[64.56 INFO] (SDKD log:137) }
[64.57 INFO] (SDKD log:137) }
[67.50 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[67.50 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@102.5
[67.51 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[67.51 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[67.51 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[67.52 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[67.52 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[67.52 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[67.52 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[67.53 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.53 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[67.53 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[67.55 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@103.8 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCou
nt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[67.56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[67.56 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3
.4.147",8091]]}, Hostname=10.3.4.144}
[67.57 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.57 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.57 INFO] (SDKD log:137) "DS": {},
[67.58 INFO] (SDKD log:137) "ViewParameters": {},
[67.58 INFO] (SDKD log:137) "DesignName": "test_design",
[67.58 INFO] (SDKD log:137) "Options": {
[67.58 INFO] (SDKD log:137) "DelayMax": 10,
[67.59 INFO] (SDKD log:137) "DelayMin": 1,
[67.59 INFO] (SDKD log:137) "IterWait": 1,
[67.60 INFO] (SDKD log:137) "ViewQueryCount": -1,
[67.60 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.9
[67.60 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[67.61 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@104.10 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[67.61 INFO] (SDKD log:137) "TimeRes": 1
[67.61 INFO] (SDKD log:137) },
[67.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[67.62 INFO] (SDKD log:137) "ViewName": "test_view"
[67.62 INFO] (SDKD log:137) }
[67.63 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.63 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[67.63 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.64 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[67.64 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.64 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.64 INFO] (SDKD log:137) "DS": {},
[67.65 INFO] (SDKD log:137) "ViewParameters": {},
[67.65 INFO] (SDKD log:137) "DesignName": "test_design",
[67.65 INFO] (SDKD log:137) "Options": {
[67.66 INFO] (SDKD log:137) "DelayMax": 10,
[67.66 INFO] (SDKD log:137) "DelayMin": 1,
[67.66 INFO] (SDKD log:137) "IterWait": 1,
[67.67 INFO] (SDKD log:137) "ViewQueryCount": -1,
[67.67 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[67.67 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.11
[67.67 INFO] (SDKD log:137) "TimeRes": 1
[67.68 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@105.12 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[67.68 INFO] (SDKD log:137) },
[67.69 INFO] (SDKD log:137) "ViewName": "test_view"
[67.69 INFO] (SDKD log:137) }
[67.69 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.13 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[67.70 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.70 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[67.70 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.71 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[67.71 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.72 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.72 INFO] (SDKD log:137) "DS": {},
[67.72 INFO] (SDKD log:137) "ViewParameters": {},
[67.73 INFO] (SDKD log:137) "DesignName": "test_design",
[67.73 INFO] (SDKD log:137) "Options": {
[67.74 INFO] (SDKD log:137) "DelayMax": 10,
[67.74 INFO] (SDKD log:137) "DelayMin": 1,
[67.74 INFO] (SDKD log:137) "IterWait": 1,
[67.75 INFO] (SDKD log:137) "ViewQueryCount": -1,
[67.75 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[67.75 INFO] (SDKD log:137) "TimeRes": 1
[67.76 INFO] (SDKD log:137) },
[67.76 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.13
[67.76 INFO] (SDKD log:137) "ViewName": "test_view"
[67.77 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@106.14 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[67.78 INFO] (SDKD log:137) }
[67.78 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.78 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[67.79 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[67.79 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.79 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[67.80 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.80 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.80 INFO] (SDKD log:137) "DS": {},
[67.81 INFO] (SDKD log:137) "ViewParameters": {},
[67.81 INFO] (SDKD log:137) "DesignName": "test_design",
[67.81 INFO] (SDKD log:137) "Options": {
[67.82 INFO] (SDKD log:137) "DelayMax": 10,
[67.82 INFO] (SDKD log:137) "DelayMin": 1,
[67.82 INFO] (SDKD log:137) "IterWait": 1,
[67.82 INFO] (SDKD log:137) "ViewQueryCount": -1,
[67.83 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.15
[67.83 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[67.83 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@107.16 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCo
unt=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[67.84 INFO] (SDKD log:137) "TimeRes": 1
[67.84 INFO] (SDKD log:137) },
[67.84 INFO] (SDKD log:137) "ViewName": "test_view"
[67.84 INFO] (Workload call:112) Preloading items into cluster before workload
[67.85 INFO] (SDKD log:137) }
[67.85 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.17 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.
3.4.147",8091]]}, Hostname=10.3.4.144}
[67.85 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.86 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[67.86 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[67.86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.87 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.87 INFO] (SDKD log:137) "DS": {},
[67.87 INFO] (SDKD log:137) "ViewParameters": {},
[67.88 INFO] (SDKD log:137) "DesignName": "test_design",
[67.88 INFO] (SDKD log:137) "Options": {
[67.88 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.17
[67.88 INFO] (SDKD log:137) "DelayMax": 10,
[67.89 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.18 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=false, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=0, DelayMin=0, IterWait=100, TimeRes=0}}
[67.89 INFO] (SDKD log:137) "DelayMin": 1,
[67.89 INFO] (SDKD log:137) "IterWait": 1,
[67.90 INFO] (SDKD log:137) "ViewQueryCount": -1,
[67.91 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[67.91 INFO] (SDKD log:137) "TimeRes": 1
[67.91 INFO] (SDKD log:137) },
[67.92 INFO] (SDKD log:137) "ViewName": "test_view"
[67.92 INFO] (SDKD log:137) }
[67.92 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[67.93 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[67.93 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[67.93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[67.93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[67.94 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[67.94 INFO] (SDKD log:137) "DS": {
[67.94 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[67.94 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[67.95 INFO] (SDKD log:137) "Count": 15000,
[67.95 INFO] (SDKD log:137) "Continuous": false,
[67.95 INFO] (SDKD log:137) "VSize": 128,
[67.96 INFO] (SDKD log:137) "Repeat": "REP",
[67.96 INFO] (SDKD log:137) "KSize": 32
[67.96 INFO] (SDKD log:137) },
[67.96 INFO] (SDKD log:137) "Options": {
[67.97 INFO] (SDKD log:137) "DelayMax": 0,
[67.97 INFO] (SDKD log:137) "DelayMin": 0,
[67.97 INFO] (SDKD log:137) "IterWait": 100,
[67.98 INFO] (SDKD log:137) "TimeRes": 0
[67.98 INFO] (SDKD log:137) }
[67.98 INFO] (SDKD log:137) }
[270.59 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@108.18 => {[OK]: 4231, [MEMD:MEMD_EVBUCKET]: 10769}
[270.59 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@108.19
[270.59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[270.60 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[270.60 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.21 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.60 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.29 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.60 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.28 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.60 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.27 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.25 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.26 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.24 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.23 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.22 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.61 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[270.61 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.20 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.62 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[270.63 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[270.64 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[270.65 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[270.65 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[270.65 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[270.65 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.65 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.66 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[270.66 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.66 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[270.66 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[270.67 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.67 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.21
[270.67 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@110.30 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.68 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.31 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.68 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[270.69 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.69 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.69 INFO] (SDKD log:137) "DS": {
[270.69 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.69 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.69 INFO] (SDKD log:137) "Count": 15000,
[270.69 INFO] (SDKD log:137) "Continuous": true,
[270.70 INFO] (SDKD log:137) "VSize": 128,
[270.70 INFO] (SDKD log:137) "Repeat": "REP",
[270.70 INFO] (SDKD log:137) "KSize": 32
[270.70 INFO] (SDKD log:137) },
[270.70 INFO] (SDKD log:137) "Options": {
[270.70 INFO] (SDKD log:137) "DelayMax": 10,
[270.71 INFO] (SDKD log:137) "DelayMin": 1,
[270.71 INFO] (SDKD log:137) "IterWait": 1,
[270.71 INFO] (SDKD log:137) "TimeRes": 1
[270.71 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@114.25
[270.71 INFO] (SDKD log:137) }
[270.72 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@114.32 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.72 INFO] (SDKD log:137) }
[270.72 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.72 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[270.73 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.73 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.33 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.73 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[270.73 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.73 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.73 INFO] (SDKD log:137) "DS": {
[270.73 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.74 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.74 INFO] (SDKD log:137) "Count": 15000,
[270.74 INFO] (SDKD log:137) "Continuous": true,
[270.74 INFO] (SDKD log:137) "VSize": 128,
[270.75 INFO] (SDKD log:137) "Repeat": "REP",
[270.75 INFO] (SDKD log:137) "KSize": 32
[270.75 INFO] (SDKD log:137) },
[270.75 INFO] (SDKD log:137) "Options": {
[270.75 INFO] (SDKD log:137) "DelayMax": 10,
[270.76 INFO] (SDKD log:137) "DelayMin": 1,
[270.76 INFO] (SDKD log:137) "IterWait": 1,
[270.76 INFO] (SDKD log:137) "TimeRes": 1
[270.76 INFO] (SDKD log:137) }
[270.76 INFO] (SDKD log:137) }
[270.77 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.77 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120
[270.77 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@115.26
[270.77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.78 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@115.34 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[270.78 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.79 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.79 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.35 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.79 INFO] (SDKD log:137) "DS": {
[270.79 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.80 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.22
[270.80 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.80 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@111.36 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.80 INFO] (SDKD log:137) "Count": 15000,
[270.80 INFO] (SDKD log:137) "Continuous": true,
[270.81 INFO] (SDKD log:137) "VSize": 128,
[270.81 INFO] (SDKD log:137) "Repeat": "REP",
[270.81 INFO] (SDKD log:137) "KSize": 32
[270.81 INFO] (SDKD log:137) },
[270.81 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@122.37 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.82 INFO] (SDKD log:137) "Options": {
[270.82 INFO] (SDKD log:137) "DelayMax": 10,
[270.82 INFO] (SDKD log:137) "DelayMin": 1,
[270.82 INFO] (SDKD log:137) "IterWait": 1,
[270.82 INFO] (SDKD log:137) "TimeRes": 1
[270.83 INFO] (SDKD log:137) }
[270.83 INFO] (SDKD log:137) }
[270.83 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.83 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.83 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121
[270.84 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[270.84 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.84 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.84 INFO] (SDKD log:137) "DS": {
[270.84 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.85 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.85 INFO] (SDKD log:137) "Count": 15000,
[270.85 INFO] (SDKD log:137) "Continuous": true,
[270.85 INFO] (SDKD log:137) "VSize": 128,
[270.85 INFO] (SDKD log:137) "Repeat": "REP",
[270.86 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.23
[270.86 INFO] (SDKD log:137) "KSize": 32
[270.86 INFO] (SDKD log:137) },
[270.86 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@112.38 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.86 INFO] (SDKD log:137) "Options": {
[270.87 INFO] (SDKD log:137) "DelayMax": 10,
[270.87 INFO] (SDKD log:137) "DelayMin": 1,
[270.87 INFO] (SDKD log:137) "IterWait": 1,
[270.87 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@123.39 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.87 INFO] (SDKD log:137) "TimeRes": 1
[270.88 INFO] (SDKD log:137) }
[270.88 INFO] (SDKD log:137) }
[270.88 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.89 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 122
[270.89 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.89 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[270.90 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.20
[270.90 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.90 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@109.40 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.90 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.91 INFO] (SDKD log:137) "DS": {
[270.91 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@124.41 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.92 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.92 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.92 INFO] (SDKD log:137) "Count": 15000,
[270.92 INFO] (SDKD log:137) "Continuous": true,
[270.92 INFO] (SDKD log:137) "VSize": 128,
[270.92 INFO] (SDKD log:137) "Repeat": "REP",
[270.93 INFO] (SDKD log:137) "KSize": 32
[270.93 INFO] (SDKD log:137) },
[270.93 INFO] (SDKD log:137) "Options": {
[270.93 INFO] (SDKD log:137) "DelayMax": 10,
[270.93 INFO] (SDKD log:137) "DelayMin": 1,
[270.94 INFO] (SDKD log:137) "IterWait": 1,
[270.94 INFO] (SDKD log:137) "TimeRes": 1
[270.94 INFO] (SDKD log:137) }
[270.94 INFO] (SDKD log:137) }
[270.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 123
[270.94 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[270.95 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[270.95 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@118.29
[270.95 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[270.96 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@118.42 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[270.96 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[270.96 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[270.97 INFO] (SDKD log:137) "DS": {
[270.97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@125.43 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[270.97 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[270.98 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[270.98 INFO] (SDKD log:137) "Count": 15000,
[270.99 INFO] (SDKD log:137) "Continuous": true,
[270.99 INFO] (SDKD log:137) "VSize": 128,
[270.99 INFO] (SDKD log:137) "Repeat": "REP",
[271.00 INFO] (SDKD log:137) "KSize": 32
[271.00 INFO] (SDKD log:137) },
[271.00 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.01 INFO] (SDKD log:137) "Options": {
[271.01 INFO] (SDKD log:137) "DelayMax": 10,
[271.01 INFO] (SDKD log:137) "DelayMin": 1,
[271.03 INFO] (SDKD log:137) "IterWait": 1,
[271.03 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.04 INFO] (SDKD log:137) "TimeRes": 1
[271.04 INFO] (SDKD log:137) }
[271.04 INFO] (SDKD log:137) }
[271.04 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 124
[271.05 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.05 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.05 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[271.05 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.06 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.06 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.06 INFO] (SDKD log:137) "DS": {
[271.06 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.06 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.07 INFO] (SDKD log:137) "Count": 15000,
[271.07 INFO] (SDKD log:137) "Continuous": true,
[271.07 INFO] (SDKD log:137) "VSize": 128,
[271.07 INFO] (SDKD log:137) "Repeat": "REP",
[271.07 INFO] (SDKD log:137) "KSize": 32
[271.07 INFO] (SDKD log:137) },
[271.08 INFO] (SDKD log:137) "Options": {
[271.08 INFO] (SDKD log:137) "DelayMax": 10,
[271.08 INFO] (SDKD log:137) "DelayMin": 1,
[271.08 INFO] (SDKD log:137) "IterWait": 1,
[271.08 INFO] (SDKD log:137) "TimeRes": 1
[271.09 INFO] (SDKD log:137) }
[271.09 INFO] (SDKD log:137) }
[271.09 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 125
[271.09 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[271.10 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@117.28
[271.10 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.10 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@117.44 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.10 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.11 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.11 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.11 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@126.45 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[271.11 INFO] (SDKD log:137) "DS": {
[271.11 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.11 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.12 INFO] (SDKD log:137) "Count": 15000,
[271.13 INFO] (SDKD log:137) "Continuous": true,
[271.13 INFO] (SDKD log:137) "VSize": 128,
[271.14 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@113.24
[271.14 INFO] (SDKD log:137) "Repeat": "REP",
[271.14 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@113.46 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.14 INFO] (SDKD log:137) "KSize": 32
[271.14 INFO] (SDKD log:137) },
[271.15 INFO] (SDKD log:137) "Options": {
[271.15 INFO] (SDKD log:137) "DelayMax": 10,
[271.15 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@127.47 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[271.15 INFO] (SDKD log:137) "DelayMin": 1,
[271.15 INFO] (SDKD log:137) "IterWait": 1,
[271.15 INFO] (SDKD log:137) "TimeRes": 1
[271.15 INFO] (SDKD log:137) }
[271.16 INFO] (SDKD log:137) }
[271.16 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[271.16 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 126
[271.17 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.17 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.17 INFO] (SDKD log:137) "DS": {
[271.17 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.17 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.17 INFO] (SDKD log:137) "Count": 15000,
[271.18 INFO] (SDKD log:137) "Continuous": true,
[271.18 INFO] (SDKD log:137) "VSize": 128,
[271.18 INFO] (SDKD log:137) "Repeat": "REP",
[271.18 INFO] (SDKD log:137) "KSize": 32
[271.18 INFO] (SDKD log:137) },
[271.18 INFO] (SDKD log:137) "Options": {
[271.19 INFO] (SDKD log:137) "DelayMax": 10,
[271.19 INFO] (SDKD log:137) "DelayMin": 1,
[271.19 INFO] (SDKD log:137) "IterWait": 1,
[271.19 INFO] (SDKD log:137) "TimeRes": 1
[271.19 INFO] (SDKD log:137) }
[271.19 INFO] (SDKD log:137) }
[271.20 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[271.20 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 127
[271.20 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.20 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@116.27
[271.20 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@116.48 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.21 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.22 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@128.49 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[271.22 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.22 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.23 INFO] (SDKD log:137) "DS": {
[271.23 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.23 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.23 INFO] (SDKD log:137) "Count": 15000,
[271.23 INFO] (SDKD log:137) "Continuous": true,
[271.23 INFO] (SDKD log:137) "VSize": 128,
[271.23 INFO] (SDKD log:137) "Repeat": "REP",
[271.23 INFO] (SDKD log:137) "KSize": 32
[271.24 INFO] (SDKD log:137) },
[271.24 INFO] (SDKD log:137) "Options": {
[271.24 INFO] (SDKD log:137) "DelayMax": 10,
[271.24 INFO] (SDKD log:137) "DelayMin": 1,
[271.24 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@119.31
[271.25 INFO] (SDKD log:137) "IterWait": 1,
[271.25 INFO] (SDKD log:137) "TimeRes": 1
[271.25 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@119.50 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.25 INFO] (SDKD log:137) }
[271.25 INFO] (SDKD log:137) }
[271.26 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[271.26 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 128
[271.26 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.26 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.26 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.26 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.27 INFO] (SDKD log:137) "DS": {
[271.27 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.27 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.27 INFO] (SDKD log:137) "Count": 15000,
[271.27 INFO] (SDKD log:137) "Continuous": true,
[271.27 INFO] (SDKD log:137) "VSize": 128,
[271.27 INFO] (SDKD log:137) "Repeat": "REP",
[271.28 INFO] (SDKD log:137) "KSize": 32
[271.28 INFO] (SDKD log:137) },
[271.28 INFO] (SDKD log:137) "Options": {
[271.28 INFO] (SDKD log:137) "DelayMax": 10,
[271.29 INFO] (SDKD log:137) "DelayMin": 1,
[271.29 INFO] (SDKD log:137) "IterWait": 1,
[271.29 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@120.33
[271.29 INFO] (SDKD log:137) "TimeRes": 1
[271.29 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@120.51 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.29 INFO] (SDKD log:137) }
[271.29 INFO] (SDKD log:137) }
[271.30 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.30 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.31 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.31 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.31 INFO] (SDKD log:137) "DS": {
[271.31 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.31 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[271.31 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.31 INFO] (SDKD log:137) at System.Net.Sockets.Socket.get_RemoteEndPoint()
   at Couchbase.IO.Strategies.EapConnection.<>c__DisplayClass3`1.<Send>b__0(FormatMessageHandler m) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line
38
   at Common.Logging.Factory.AbstractLogger.FormatMessageCallbackFormattedMessage.ToString()
   at log4net.ObjectRenderer.DefaultRenderer.RenderObject(RendererMap rendererMap, Object obj, TextWriter writer)
   at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj, TextWriter writer)
[271.32 INFO] (SDKD log:137) "Count": 15000,
[271.32 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.32 INFO] (SDKD log:137) "Continuous": true,
[271.33 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.33 INFO] (SDKD log:137) "VSize": 128,
[271.33 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.33 INFO] (SDKD log:137) "Repeat": "REP",
[271.33 INFO] (SDKD log:137) "KSize": 32
[271.33 INFO] (SDKD log:137) },
[271.34 INFO] (SDKD log:137) "Options": {
[271.34 INFO] (SDKD log:137) "DelayMax": 10,
[271.34 INFO] (SDKD log:137) "DelayMin": 1,
[271.34 INFO] (SDKD log:137) "IterWait": 1,
[271.34 INFO] (SDKD log:137) "TimeRes": 1
[271.34 INFO] (SDKD log:137) }
[271.34 INFO] (SDKD log:137) }
[271.35 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.35 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.35 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.36 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.38 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.38 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@121.35
[271.39 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@121.52 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.39 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.39 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.39 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.39 INFO] (SDKD log:137) "DS": {
[271.40 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.40 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.40 INFO] (SDKD log:137) "Count": 15000,
[271.40 INFO] (SDKD log:137) "Continuous": true,
[271.40 INFO] (SDKD log:137) "VSize": 128,
[271.41 INFO] (SDKD log:137) "Repeat": "REP",
[271.41 INFO] (SDKD log:137) "KSize": 32
[271.41 INFO] (SDKD log:137) },
[271.41 INFO] (SDKD log:137) "Options": {
[271.41 INFO] (SDKD log:137) "DelayMax": 10,
[271.41 INFO] (SDKD log:137) "DelayMin": 1,
[271.41 INFO] (SDKD log:137) "IterWait": 1,
[271.41 INFO] (SDKD log:137) "TimeRes": 1
[271.42 INFO] (SDKD log:137) }
[271.42 INFO] (SDKD log:137) }
[271.47 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.47 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@122.37
[271.48 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@122.53 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.48 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.48 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.48 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.49 INFO] (SDKD log:137) "DS": {
[271.49 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.51 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.51 INFO] (SDKD log:137) "Count": 15000,
[271.51 INFO] (SDKD log:137) "Continuous": true,
[271.51 INFO] (SDKD log:137) "VSize": 128,
[271.51 INFO] (SDKD log:137) "Repeat": "REP",
[271.51 INFO] (SDKD log:137) "KSize": 32
[271.52 INFO] (SDKD log:137) },
[271.52 INFO] (SDKD log:137) "Options": {
[271.52 INFO] (SDKD log:137) "DelayMax": 10,
[271.52 INFO] (SDKD log:137) "DelayMin": 1,
[271.52 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.52 INFO] (SDKD log:137) "IterWait": 1,
[271.52 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.53 INFO] (SDKD log:137) "TimeRes": 1
[271.53 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.53 INFO] (SDKD log:137) }
[271.53 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.53 INFO] (SDKD log:137) }
[271.55 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@123.39
[271.55 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@123.54 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.56 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.56 INFO] (SDKD log:137) "DS": {
[271.56 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.56 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.56 INFO] (SDKD log:137) "Count": 15000,
[271.56 INFO] (SDKD log:137) "Continuous": true,
[271.57 INFO] (SDKD log:137) "VSize": 128,
[271.57 INFO] (SDKD log:137) "Repeat": "REP",
[271.57 INFO] (SDKD log:137) "KSize": 32
[271.57 INFO] (SDKD log:137) },
[271.57 INFO] (SDKD log:137) "Options": {
[271.58 INFO] (SDKD log:137) "DelayMax": 10,
[271.58 INFO] (SDKD log:137) "DelayMin": 1,
[271.58 INFO] (SDKD log:137) "IterWait": 1,
[271.58 INFO] (SDKD log:137) "TimeRes": 1
[271.58 INFO] (SDKD log:137) }
[271.58 INFO] (SDKD log:137) }
[271.61 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.62 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@124.41
[271.62 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@124.55 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.63 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.63 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.63 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.63 INFO] (SDKD log:137) "DS": {
[271.63 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.64 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.64 INFO] (SDKD log:137) "Count": 15000,
[271.64 INFO] (SDKD log:137) "Continuous": true,
[271.64 INFO] (SDKD log:137) "VSize": 128,
[271.64 INFO] (SDKD log:137) "Repeat": "REP",
[271.64 INFO] (SDKD log:137) "KSize": 32
[271.64 INFO] (SDKD log:137) },
[271.65 INFO] (SDKD log:137) "Options": {
[271.65 INFO] (SDKD log:137) "DelayMax": 10,
[271.65 INFO] (SDKD log:137) "DelayMin": 1,
[271.65 INFO] (SDKD log:137) "IterWait": 1,
[271.65 INFO] (SDKD log:137) "TimeRes": 1
[271.65 INFO] (SDKD log:137) }
[271.65 INFO] (SDKD log:137) }
[271.67 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.67 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.67 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.67 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.68 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.68 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.68 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.68 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.68 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.68 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.68 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.69 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.69 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.69 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.69 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.69 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.69 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.69 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.71 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.72 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.73 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.73 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.75 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.76 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.76 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.78 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.79 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.79 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.79 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@125.43
[271.79 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.80 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@125.56 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.80 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[271.80 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.80 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.81 INFO] (SDKD log:137) "DS": {
[271.81 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.81 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.82 INFO] (SDKD log:137) "Count": 15000,
[271.82 INFO] (SDKD log:137) "Continuous": true,
[271.82 INFO] (SDKD log:137) "VSize": 128,
[271.82 INFO] (SDKD log:137) "Repeat": "REP",
[271.82 INFO] (SDKD log:137) "KSize": 32
[271.82 INFO] (SDKD log:137) },
[271.82 INFO] (SDKD log:137) "Options": {
[271.82 INFO] (SDKD log:137) "DelayMax": 10,
[271.83 INFO] (SDKD log:137) "DelayMin": 1,
[271.83 INFO] (SDKD log:137) "IterWait": 1,
[271.83 INFO] (SDKD log:137) "TimeRes": 1
[271.83 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@126.45
[271.83 INFO] (SDKD log:137) }
[271.84 INFO] (SDKD log:137) }
[271.84 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@126.57 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[271.84 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[271.85 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[271.86 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.87 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[271.87 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.87 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[271.87 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.87 INFO] (SDKD log:137) "DS": {
[271.87 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.87 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[271.88 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[271.88 INFO] (SDKD log:137) "Count": 15000,
[271.88 INFO] (SDKD log:137) "Continuous": true,
[271.88 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.88 INFO] (SDKD log:137) "VSize": 128,
[271.88 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.88 INFO] (SDKD log:137) "Repeat": "REP",
[271.88 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.89 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.89 INFO] (SDKD log:137) "KSize": 32
[271.89 INFO] (SDKD log:137) },
[271.89 INFO] (SDKD log:137) "Options": {
[271.90 INFO] (SDKD log:137) "DelayMax": 10,
[271.90 INFO] (SDKD log:137) "DelayMin": 1,
[271.90 INFO] (SDKD log:137) "IterWait": 1,
[271.90 INFO] (SDKD log:137) "TimeRes": 1
[271.90 INFO] (SDKD log:137) }
[271.90 INFO] (SDKD log:137) }
[271.91 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.92 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.92 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.92 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.92 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.92 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.93 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.93 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.93 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.93 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.93 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.93 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.94 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.94 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.95 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.95 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.96 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.96 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.96 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.96 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.97 INFO] (SDKD log:137) +++ Received exception of ID 0
[271.97 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[271.97 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[271.97 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[271.99 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[272.00 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@127.47
[272.00 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@127.58 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32},
 Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[272.01 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.01 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.01 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.02 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.02 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.02 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.02 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.02 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.03 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.03 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.03 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.03 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.03 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.04 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.04 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.04 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[272.04 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.04 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[272.04 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.04 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[272.04 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.04 INFO] (SDKD log:137) "DS": {
[272.05 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.05 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[272.05 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.05 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[272.05 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.05 INFO] (SDKD log:137) "Count": 15000,
[272.05 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.05 INFO] (SDKD log:137) "Continuous": true,
[272.05 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.06 INFO] (SDKD log:137) "VSize": 128,
[272.06 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.07 INFO] (SDKD log:137) "Repeat": "REP",
[272.07 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.07 INFO] (SDKD log:137) "KSize": 32
[272.07 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.07 INFO] (SDKD log:137) },
[272.07 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.07 INFO] (SDKD log:137) "Options": {
[272.07 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.07 INFO] (SDKD log:137) "DelayMax": 10,
[272.08 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.08 INFO] (SDKD log:137) "DelayMin": 1,
[272.08 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.08 INFO] (SDKD log:137) "IterWait": 1,
[272.08 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.08 INFO] (SDKD log:137) "TimeRes": 1
[272.08 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.09 INFO] (SDKD log:137) }
[272.09 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.09 INFO] (SDKD log:137) }
[272.09 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.09 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.10 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.10 INFO] (SDKD log:137) +++ Received exception of ID 0
[272.10 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[272.11 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[272.11 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[272.15 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@128.49
[272.16 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@128.59 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Option
s={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[272.17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[272.17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[272.17 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@129.60 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[272.17 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[272.18 INFO] (SDKD log:137) "DS": {
[272.18 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[272.18 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[272.19 INFO] (SDKD log:137) "Count": 15000,
[272.19 INFO] (SDKD log:137) "Continuous": true,
[272.19 INFO] (SDKD log:137) "VSize": 128,
[272.19 INFO] (SDKD log:137) "Repeat": "REP",
[272.19 INFO] (SDKD log:137) "KSize": 32
[272.20 INFO] (SDKD log:137) },
[272.20 INFO] (SDKD log:137) "Options": {
[272.20 INFO] (SDKD log:137) "DelayMax": 10,
[272.20 INFO] (SDKD log:137) "DelayMin": 1,
[272.20 INFO] (SDKD log:137) "IterWait": 1,
[272.21 INFO] (SDKD log:137) "TimeRes": 1
[272.21 INFO] (SDKD log:137) }
[272.21 INFO] (SDKD log:137) }
[272.21 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[272.22 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 129
[272.22 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[272.24 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@129.60
[272.24 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@129.61 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=
12}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[272.25 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[272.25 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[272.26 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[272.26 INFO] (SDKD log:137) "DS": {
[272.26 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[272.26 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[272.27 INFO] (SDKD log:137) "Count": 1000,
[272.27 INFO] (SDKD log:137) "Continuous": false,
[272.27 INFO] (SDKD log:137) "VSize": 512,
[272.27 INFO] (SDKD log:137) "Repeat": "rep",
[272.27 INFO] (SDKD log:137) "KSize": 12
[272.27 INFO] (SDKD log:137) },
[272.28 INFO] (SDKD log:137) "Schema": {
[272.28 INFO] (SDKD log:137) "InflateContent": "meh",
[272.28 INFO] (SDKD log:137) "InflateLevel": 40
[272.28 INFO] (SDKD log:137) },
[272.28 INFO] (SDKD log:137) "Options": {
[272.28 INFO] (SDKD log:137) "DelayMax": 0,
[272.28 INFO] (SDKD log:137) "DelayMin": 0,
[272.29 INFO] (SDKD log:137) "IterWait": 1,
[272.29 INFO] (SDKD log:137) "TimeRes": 0
[272.29 INFO] (SDKD log:137) }
[272.29 INFO] (SDKD log:137) }
[439.04 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[439.04 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@129.61
[439.05 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@129.62
[439.05 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[439.05 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@130.63 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[439.05 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[439.05 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1303659411
[439.06 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1303659411
[439.06 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[439.06 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[439.06 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 130
[439.06 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[439.11 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@130.63
[439.11 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@130.64 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[439.12 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@131.65 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[439.12 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[439.13 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[439.13 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[439.13 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[439.13 INFO] (SDKD log:137) "DS": {},
[439.14 INFO] (SDKD log:137) "ViewParameters": {
[439.14 INFO] (SDKD log:137) "limit": 0
[439.14 INFO] (SDKD log:137) },
[439.14 INFO] (SDKD log:137) "DesignName": "test_design",
[439.15 INFO] (SDKD log:137) "Options": {
[439.15 INFO] (SDKD log:137) "DelayMax": 10,
[439.15 INFO] (SDKD log:137) "DelayMin": 1,
[439.15 INFO] (SDKD log:137) "IterWait": 1,
[439.16 INFO] (SDKD log:137) "ViewQueryCount": -1,
[439.16 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[439.16 INFO] (SDKD log:137) "TimeRes": 1
[439.17 INFO] (SDKD log:137) },
[439.17 INFO] (SDKD log:137) "ViewName": "test_view"
[439.17 INFO] (SDKD log:137) }
[439.17 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 131
[439.17 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[439.21 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@131.65
[439.21 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@131.66 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[439.21 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[439.22 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[439.22 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@132.67 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[439.22 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[439.22 INFO] (SDKD log:137) "DS": {},
[439.23 INFO] (SDKD log:137) "ViewParameters": {
[439.23 INFO] (SDKD log:137) "limit": 0
[439.23 INFO] (SDKD log:137) },
[439.23 INFO] (SDKD log:137) "DesignName": "test_design",
[439.23 INFO] (SDKD log:137) "Options": {
[439.23 INFO] (SDKD log:137) "DelayMax": 10,
[439.24 INFO] (SDKD log:137) "DelayMin": 1,
[439.24 INFO] (SDKD log:137) "IterWait": 1,
[439.24 INFO] (SDKD log:137) "ViewQueryCount": -1,
[439.24 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[439.25 INFO] (SDKD log:137) "TimeRes": 1
[439.25 INFO] (SDKD log:137) },
[439.25 INFO] (SDKD log:137) "ViewName": "test_view"
[439.25 INFO] (SDKD log:137) }
[439.25 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[439.26 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 132
[439.26 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[439.28 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@132.67
[439.28 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@132.68 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[439.29 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[439.29 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@133.69 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[439.29 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[439.29 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[439.30 INFO] (SDKD log:137) "DS": {},
[439.30 INFO] (SDKD log:137) "ViewParameters": {
[439.30 INFO] (SDKD log:137) "limit": 0
[439.30 INFO] (SDKD log:137) },
[439.30 INFO] (SDKD log:137) "DesignName": "test_design",
[439.30 INFO] (SDKD log:137) "Options": {
[439.31 INFO] (SDKD log:137) "DelayMax": 10,
[439.31 INFO] (SDKD log:137) "DelayMin": 1,
[439.31 INFO] (SDKD log:137) "IterWait": 1,
[439.31 INFO] (SDKD log:137) "ViewQueryCount": -1,
[439.32 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[439.32 INFO] (SDKD log:137) "TimeRes": 1
[439.32 INFO] (SDKD log:137) },
[439.32 INFO] (SDKD log:137) "ViewName": "test_view"
[439.32 INFO] (SDKD log:137) }
[439.32 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[439.33 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 133
[439.33 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[439.38 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@133.69
[439.38 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@133.70 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[439.39 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[439.39 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@134.71 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10
.3.4.147",8091]]}, Hostname=10.3.4.144}
[439.39 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[439.40 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[439.40 INFO] (SDKD log:137) "DS": {},
[439.40 INFO] (SDKD log:137) "ViewParameters": {
[439.40 INFO] (SDKD log:137) "limit": 0
[439.40 INFO] (SDKD log:137) },
[439.41 INFO] (SDKD log:137) "DesignName": "test_design",
[439.41 INFO] (SDKD log:137) "Options": {
[439.41 INFO] (SDKD log:137) "DelayMax": 10,
[439.41 INFO] (SDKD log:137) "DelayMin": 1,
[439.41 INFO] (SDKD log:137) "IterWait": 1,
[439.41 INFO] (SDKD log:137) "ViewQueryCount": -1,
[439.42 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[439.42 INFO] (SDKD log:137) "TimeRes": 1
[439.42 INFO] (SDKD log:137) },
[439.42 INFO] (SDKD log:137) "ViewName": "test_view"
[439.42 INFO] (SDKD log:137) }
[439.42 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[439.43 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 134
[439.43 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[439.43 INFO] (SDKD log:137) +++ Received exception of ID 0
[439.44 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[439.44 INFO] (SDKD log:137) System.ObjectDisposedException: Cannot access a disposed object.
[439.44 INFO] (SDKD log:137) Object name: 'System.Net.Sockets.Socket'.
[439.46 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@134.71
[439.47 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@134.72 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={limit=0}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, Vie
wQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[439.48 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[439.50 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[439.50 INFO] (Scenario run:83) RAMP phase started. Waiting for 30 seconds
[439.51 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[439.51 INFO] (SDKD log:137) "DS": {},
[439.51 INFO] (SDKD log:137) "ViewParameters": {
[439.51 INFO] (SDKD log:137) "limit": 0
[439.51 INFO] (SDKD log:137) },
[439.52 INFO] (SDKD log:137) "DesignName": "test_design",
[439.52 INFO] (SDKD log:137) "Options": {
[439.52 INFO] (SDKD log:137) "DelayMax": 10,
[439.52 INFO] (SDKD log:137) "DelayMin": 1,
[439.52 INFO] (SDKD log:137) "IterWait": 1,
[439.52 INFO] (SDKD log:137) "ViewQueryCount": -1,
[439.52 INFO] (SDKD log:137) "ViewQueryDelay": 0,
[439.53 INFO] (SDKD log:137) "TimeRes": 1
[439.53 INFO] (SDKD log:137) },
[439.53 INFO] (SDKD log:137) "ViewName": "test_view"
[439.53 INFO] (SDKD log:137) }
[469.51 INFO] (Scenario run:86) Starting CHANGE phase
[469.56 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[470.58 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[471.59 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[472.62 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[473.65 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 1%
[474.66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[475.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[476.69 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[477.70 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3%
[478.71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 4%
[479.71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 5%
[480.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 5%
[481.74 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 6%
[482.76 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 6%
[483.78 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 7%
[484.79 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 7%
[485.80 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 8%
[486.81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 9%
[487.82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 9%
[488.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10%
[489.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10%
[490.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 11%
[491.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 12%
[492.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 12%
[493.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 13%
[494.92 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 14%
[495.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15%
[496.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15%
[497.95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 16%
[498.95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[499.96 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[500.97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[501.97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[502.98 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[503.99 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[505.00 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[506.01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 19%
[507.01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 19%
[508.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 20%
[509.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 20%
[510.04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 20%
[511.05 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 21%
[512.07 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 22%
[513.08 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 23%
[514.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 25%
[515.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 26%
[516.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 28%
[517.11 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 29%
[518.13 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 30%
[519.15 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 31%
[520.15 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 31%
[521.17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32%
[522.18 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 32%
[523.19 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 33%
[524.20 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 33%
[525.34 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 34%
[526.35 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 35%
[527.36 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 36%
[528.37 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 37%
[529.38 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 38%
[530.64 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 38%
[531.65 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 39%
[532.67 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 40%
[533.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 40%
[534.70 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 41%
[535.71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 41%
[536.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 42%
[537.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 43%
[538.74 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 44%
[539.75 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 44%
[540.76 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 45%
[541.77 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 46%
[542.78 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 48%
[543.79 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 49%
[544.81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 50%
[545.82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 51%
[546.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 52%
[547.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 53%
[548.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 55%
[549.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[551.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[552.39 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 56%
[553.41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 57%
[554.42 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 57%
[555.49 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 58%
[556.50 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 58%
[557.51 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 59%
[558.53 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 60%
[559.54 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 61%
[560.59 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 61%
[561.61 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 62%
[562.61 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 63%
[563.62 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 64%
[564.63 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 64%
[565.65 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 65%
[566.66 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 65%
[567.67 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 66%
[568.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 67%
[569.69 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 67%
[570.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 68%
[571.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 68%
[572.73 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 69%
[573.75 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 69%
[574.76 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 70%
[575.77 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 70%
[576.78 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 71%
[577.79 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 71%
[578.81 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 72%
[579.82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 73%
[580.82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 73%
[581.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 74%
[582.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 74%
[583.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 75%
[584.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 75%
[585.87 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 75%
[587.26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 76%
[588.74 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 76%
[590.00 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[591.01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[592.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[593.04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[594.07 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[595.08 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 77%
[596.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 78%
[597.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 78%
[599.20 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 78%
[600.21 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[601.22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[602.23 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[603.24 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[604.25 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[605.26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[606.27 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 79%
[607.28 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 80%
[608.29 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 80%
[609.30 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 80%
[610.32 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 80%
[611.33 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 80%
[612.34 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[613.35 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[614.36 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[615.38 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[616.39 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[617.41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[618.41 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[619.42 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[620.46 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 81%
[621.47 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[622.47 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[623.48 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[625.72 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[626.75 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 82%
[627.76 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[628.77 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[629.80 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[630.82 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 83%
[631.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 84%
[632.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 84%
[633.85 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[634.86 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[635.90 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[636.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 85%
[637.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 86%
[638.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 86%
[639.93 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[640.94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[641.95 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 87%
[642.96 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 88%
[643.97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 88%
[644.97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[645.98 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[646.99 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 89%
[648.00 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[649.00 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[650.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[651.02 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 90%
[652.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[653.03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[654.04 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[655.05 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 91%
[656.06 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[657.07 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[658.07 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 92%
[659.08 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 93%
[660.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 93%
[661.09 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 94%
[662.10 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 96%
[662.81 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace-n
et2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[663.11 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 97%
[664.12 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 99%
[665.13 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[666.14 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[667.15 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[668.16 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[669.17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[670.19 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[671.47 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[672.48 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[673.49 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[674.50 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[675.52 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[675.52 INFO] (Scenario run:97) CHANGE phase done
[675.52 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[676.23 INFO] (SDKD log:137) +++ Received exception of ID 0
[676.23 INFO] (SDKD log:137)
[676.23 INFO] (SDKD log:137) Unhandled Exception:
[676.24 INFO] (SDKD log:137) Unhandled Exception:
[676.24 INFO] (SDKD log:137)
[676.24 INFO] (SDKD log:137)
[676.24 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception: Unhandled Exception:
[676.25 INFO] (SDKD log:137)
[676.25 INFO] (SDKD log:137) Unhandled Exception:
[676.26 INFO] (SDKD log:137) Unhandled Exception:
[676.26 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception:
[676.26 INFO] (SDKD log:137)
[676.26 INFO] (SDKD log:137)
[676.26 INFO] (SDKD log:137) Unhandled Exception:
[676.26 INFO] (SDKD log:137)
[676.26 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception:
[676.27 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.27 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3
[676.27 INFO] (SDKD log:137) at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 172
   at Couchbase.Core.Buckets.CouchbaseBucket.HandleIOError[T](IOperation`1 operation, IServer server) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:li
ne 194
   at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, IServer server) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couc
hbase\Core\Buckets\CouchbaseBucket.cs:line 179
   at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 104
   at Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 312
   at Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:line 77
   at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[676.28 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.28 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.28 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.28 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 5
[676.28 INFO] (SDKD log:137) at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 172
   at Couchbase.Core.Buckets.CouchbaseBucket.HandleIOError[T](IOperation`1 operation, IServer server) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:li
ne 194
   at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, IServer server) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couc
hbase\Core\Buckets\CouchbaseBucket.cs:line 179
   at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 104
   at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 888
   at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\GetCommand.cs:line 18
   at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[676.29 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.29 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.29 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.30 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.30 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.30 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.30 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.30 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.30 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.31 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.31 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.31 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.31 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.31 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.32 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.32 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.32 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.32 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.32 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.32 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.32 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.33 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.33 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.33 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.33 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.33 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.33 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.34 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.34 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.34 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.34 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.34 INFO] (SDKD log:137) +++ Received exception of ID 2
[676.34 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.35 INFO] (SDKD log:137) +++ Received exception of ID 4
[676.36 INFO] (SDKD log:137)
[676.36 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.49 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 7
[676.49 INFO] (SDKD log:137) at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 172
   at Couchbase.Core.Buckets.CouchbaseBucket.CreateQuery(Boolean development, String designdoc, String viewname) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseB
ucket.cs:line 1165
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 161
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[676.50 INFO] (SDKD log:137)
[676.50 INFO] (SDKD log:137)
[676.50 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception: Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.50 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.50 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.50 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.50 INFO] (SDKD log:137) System.InvalidOperationException: Sequence contains no matching element
[676.51 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.51 INFO] (SDKD log:137)
[676.51 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.51 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.51 INFO] (SDKD log:137)
[676.51 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.51 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.51 INFO] (SDKD log:137)
[676.52 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.52 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.52 INFO] (SDKD log:137)
[676.52 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.53 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.53 INFO] (SDKD log:137)
[676.53 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[676.53 INFO] (SDKD log:137) +++ Received exception of ID 6
[676.53 INFO] (SDKD log:137)
[676.53 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Sequence contains no matching element
[765.52 INFO] (Scenario run:105) REBOUND done. Will collect results
[765.52 DEBUG] (Handle sendMessageAsync:183) > CANCEL@103.72
[871.65 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 103
[871.65 DEBUG] (Handle receiveMessage:158) < CANCEL@103.72
[871.65 DEBUG] (Handle sendMessageAsync:183) > CANCEL@104.72
[872.57 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[872.58 INFO] (SDKD log:137) at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 172
   at Couchbase.Core.Buckets.CouchbaseBucket.CreateQuery(Boolean development, String designdoc, String viewname) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseB
ucket.cs:line 1165
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 161
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[872.60 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: java.net.SocketException: Connection reset
        at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:115) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:169) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:165) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
        at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[na:1.6.0_27]
        at java.io.InputStreamReader.read(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.fill(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:152) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.sendMessage(Handle.java:173) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.cancel(Handle.java:271) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.cancelCurrent(Handle.java:283) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.Workload.cancel(Workload.java:110) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.WorkloadGroup.cancel(WorkloadGroup.java:63) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:112) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        ... 10 common frames omitted
[872.65 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.73
[872.65 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]



 Comments   
Comment by Jeff Morris [ 16/Aug/14 ]
This should be fixed here: git fetch ssh://jmorris@review.couchbase.org:29418/couchbase-net-client refs/changes/46/40546/1 && git checkout FETCH_HEAD
Comment by Jeff Morris [ 19/Aug/14 ]
Test against github, please. The changeset has been merged already.
Comment by Andrei Baranouski [ 27/Aug/14 ]
fixed




[NCBC-587] Cannot access a disposed object in DefaultIOStrategy.Finalize() Created: 16/Aug/14  Updated: 27/Aug/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1105

Attachments: Zip Archive NCBC-587.log.zip    
Issue Links:
Gantt: finish-start

 Description   
maybe it's the continuation of NCBC-575


I took a few commits ahead from gerrit


commit 2c00a9d3a397f5c1eddc56238cf4a19e6e6618dd
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 11 18:41:08 2014 -0700

    Add config files to tests

    Change-Id: Ie093eac1da4ffc742796ffd4df8a9c63cc1c9c2f

commit 5b18623a7c13fb16f21476fe77eaf9a192e92eb2
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Mon Aug 11 17:20:19 2014 -0700

    Complete IBucket implementation for Memcached and Couchbase Buckets

    Change-Id: If6fb47b9e4a5201e625f155dadf45696b8666674

commit a6001260b0437a9779f7b7730e857b77ac9aea31
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Aug 6 15:31:06 2014 -0700

    NCBC-542: Provide Observe method on IBucket

    Change-Id: Iecfbc02490b9ce237d6fbe16619e5aa252566bd7

commit 321940886cffcdc30225201ea243d8c6e4df6da4
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Aug 6 12:31:12 2014 -0700

    NCBC-576: Do not retry operation when unknown status is returned.

    Change-Id: Ic9afeb1373d9e7804cad3c2df0521b162956326d
    Reviewed-on: http://review.couchbase.org/40347
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Matt Ingenthron <matt@couchbase.com>




c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,12 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/count=2
    workload=HybridWorkloadGroup
    rebalance/mode=out
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-16-14\025009\de7a81baca576c481ce29a22fed68ed4

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-count 2 --workload HybridWorkloadGroup
 \ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1,23 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1,48 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,49 INFO] (RunContext run:124) Starting cluster and driver
[1,49 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1,49 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,49 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[1,68 INFO] (SDKD log:137) --- Logging Self-Test ---
[1,68 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[1,68 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[1,68 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[1,68 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[1,69 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,04 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,04 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,22 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[8,65 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.145:8091,ns_1@10.3.4.145>
[9,10 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.146:8091,ns_1@10.3.4.146>
[9,56 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[10,04 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[10,05 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,05 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,05 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[12,16 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[12,16 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[12,17 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[12,17 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[12,18 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[12,18 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[12,18 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[12,19 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[12,90 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@4b54d870
[12,90 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@7f7d4e51
[12,90 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@40c718fc
[12,91 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@16c96e22
[12,91 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@3e598bc2
[27,29 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[40,62 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[49,85 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[58,58 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[59,47 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[59,47 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[59,69 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[59,92 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[67,00 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[69,58 INFO] (RunContext run:143) Driver and cluster initialized
[69,82 INFO] (RunContext call:167) Running scenario..
[69,82 INFO] (Scenario run:72) Starting RAMP phase
[69,82 INFO] (Workload setupDesign:63) Creating design test_design
[70,98 INFO] (Workload setupDesign:80) Design creation done
[70,98 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[70,98 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[70,99 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[71,00 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[71,02 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[71,02 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[71,05 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[71,05 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[78,07 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[78,07 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {Options={DelayMin=0, TimeRes=0, IterWait=1, DelayMax=0}, Schema={InflateLevel=40, InflateContent=meh}, DSType=DSTYPE_SEEDED, DS={KSize=12, KSeed=ViewFillerSeed, VSize=512, VSeed=Vie
wFillerSeed, Continuous=false, Repeat=rep, Count=1000}}
[78,08 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[78,08 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[78,08 INFO] (SDKD log:137) "Options": {
[78,09 INFO] (SDKD log:137) "DelayMin": 0,
[78,09 INFO] (SDKD log:137) "TimeRes": 0,
[78,09 INFO] (SDKD log:137) "IterWait": 1,
[78,09 INFO] (SDKD log:137) "DelayMax": 0
[78,10 INFO] (SDKD log:137) },
[78,10 INFO] (SDKD log:137) "Schema": {
[78,10 INFO] (SDKD log:137) "InflateLevel": 40,
[78,11 INFO] (SDKD log:137) "InflateContent": "meh"
[78,11 INFO] (SDKD log:137) },
[78,11 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[78,11 INFO] (SDKD log:137) "DS": {
[78,12 INFO] (SDKD log:137) "KSize": 12,
[78,12 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[78,12 INFO] (SDKD log:137) "VSize": 512,
[78,13 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[78,13 INFO] (SDKD log:137) "Continuous": false,
[78,13 INFO] (SDKD log:137) "Repeat": "rep",
[78,13 INFO] (SDKD log:137) "Count": 1000
[78,14 INFO] (SDKD log:137) }
[78,14 INFO] (SDKD log:137) }
[293,46 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[293,46 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[293,46 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[293,47 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[293,47 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[293,47 INFO] (Workload call:112) Preloading items into cluster before workload
[293,47 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[293,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[293,48 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[293,49 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[293,49 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[293,49 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[293,49 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[298,80 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[298,80 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@102.5 => {Options={DelayMin=0, TimeRes=0, IterWait=100, DelayMax=0}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=false, Repeat=REP, Coun
t=15000}}
[298,81 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[298,81 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[298,81 INFO] (SDKD log:137) "Options": {
[298,81 INFO] (SDKD log:137) "DelayMin": 0,
[298,82 INFO] (SDKD log:137) "TimeRes": 0,
[298,82 INFO] (SDKD log:137) "IterWait": 100,
[298,82 INFO] (SDKD log:137) "DelayMax": 0
[298,82 INFO] (SDKD log:137) },
[298,82 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[298,83 INFO] (SDKD log:137) "DS": {
[298,83 INFO] (SDKD log:137) "KSize": 32,
[298,83 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[298,83 INFO] (SDKD log:137) "VSize": 128,
[298,84 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[298,84 INFO] (SDKD log:137) "Continuous": false,
[298,84 INFO] (SDKD log:137) "Repeat": "REP",
[298,85 INFO] (SDKD log:137) "Count": 15000
[298,85 INFO] (SDKD log:137) }
[298,85 INFO] (SDKD log:137) }
[554,47 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[1781,34 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@102.5 => {[OK]: 14984, [MEMD:GENERIC]: 16}
[1781,34 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[1781,34 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[1781,35 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[1781,35 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.9 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,35 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,35 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.8 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.16 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.15 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.14 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.13 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.12 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,37 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.11 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,37 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.10 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1781,37 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[1781,38 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[1781,38 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[1781,38 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,38 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,38 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[1781,39 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1781,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[1781,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[1781,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1781,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[1781,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[1781,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[1781,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[1781,41 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[1781,41 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[1781,41 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[1787,97 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1787,97 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.9
[1787,98 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@105.17 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1787,98 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1787,98 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1787,98 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.18 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1787,99 INFO] (SDKD log:137) "Options": {
[1787,99 INFO] (SDKD log:137) "DelayMin": 1,
[1787,99 INFO] (SDKD log:137) "TimeRes": 1,
[1787,99 INFO] (SDKD log:137) "IterWait": 1,
[1787,99 INFO] (SDKD log:137) "DelayMax": 10
[1788,00 INFO] (SDKD log:137) },
[1788,00 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1788,00 INFO] (SDKD log:137) "DS": {
[1788,01 INFO] (SDKD log:137) "KSize": 32,
[1788,01 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1788,01 INFO] (SDKD log:137) "VSize": 128,
[1788,01 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1788,02 INFO] (SDKD log:137) "Continuous": true,
[1788,02 INFO] (SDKD log:137) "Repeat": "REP",
[1788,02 INFO] (SDKD log:137) "Count": 15000
[1788,03 INFO] (SDKD log:137) }
[1788,03 INFO] (SDKD log:137) }
[1788,03 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1788,03 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[1793,55 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1793,56 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[1793,56 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@103.19 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1793,56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1793,56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1793,57 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.20 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1793,57 INFO] (SDKD log:137) "Options": {
[1793,57 INFO] (SDKD log:137) "DelayMin": 1,
[1793,57 INFO] (SDKD log:137) "TimeRes": 1,
[1793,58 INFO] (SDKD log:137) "IterWait": 1,
[1793,58 INFO] (SDKD log:137) "DelayMax": 10
[1793,58 INFO] (SDKD log:137) },
[1793,58 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1793,58 INFO] (SDKD log:137) "DS": {
[1793,59 INFO] (SDKD log:137) "KSize": 32,
[1793,59 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1793,59 INFO] (SDKD log:137) "VSize": 128,
[1793,59 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1793,60 INFO] (SDKD log:137) "Continuous": true,
[1793,60 INFO] (SDKD log:137) "Repeat": "REP",
[1793,60 INFO] (SDKD log:137) "Count": 15000
[1793,60 INFO] (SDKD log:137) }
[1793,60 INFO] (SDKD log:137) }
[1793,61 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1793,61 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[1799,53 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1799,54 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.8
[1799,54 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@104.21 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1799,55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1799,55 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1799,55 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.22 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1799,55 INFO] (SDKD log:137) "Options": {
[1799,56 INFO] (SDKD log:137) "DelayMin": 1,
[1799,56 INFO] (SDKD log:137) "TimeRes": 1,
[1799,56 INFO] (SDKD log:137) "IterWait": 1,
[1799,56 INFO] (SDKD log:137) "DelayMax": 10
[1799,56 INFO] (SDKD log:137) },
[1799,57 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1799,57 INFO] (SDKD log:137) "DS": {
[1799,58 INFO] (SDKD log:137) "KSize": 32,
[1799,58 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1799,58 INFO] (SDKD log:137) "VSize": 128,
[1799,58 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1799,59 INFO] (SDKD log:137) "Continuous": true,
[1799,59 INFO] (SDKD log:137) "Repeat": "REP",
[1799,59 INFO] (SDKD log:137) "Count": 15000
[1799,60 INFO] (SDKD log:137) }
[1799,60 INFO] (SDKD log:137) }
[1799,60 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1799,61 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[1804,71 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1804,71 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.16
[1804,72 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@112.23 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1804,72 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1804,72 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1804,72 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.24 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1804,73 INFO] (SDKD log:137) "Options": {
[1804,73 INFO] (SDKD log:137) "DelayMin": 1,
[1804,73 INFO] (SDKD log:137) "TimeRes": 1,
[1804,73 INFO] (SDKD log:137) "IterWait": 1,
[1804,74 INFO] (SDKD log:137) "DelayMax": 10
[1804,74 INFO] (SDKD log:137) },
[1804,74 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1804,74 INFO] (SDKD log:137) "DS": {
[1804,74 INFO] (SDKD log:137) "KSize": 32,
[1804,75 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1804,75 INFO] (SDKD log:137) "VSize": 128,
[1804,75 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1804,75 INFO] (SDKD log:137) "Continuous": true,
[1804,76 INFO] (SDKD log:137) "Repeat": "REP",
[1804,76 INFO] (SDKD log:137) "Count": 15000
[1804,76 INFO] (SDKD log:137) }
[1804,76 INFO] (SDKD log:137) }
[1804,77 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1804,77 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[1809,89 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1809,89 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.15
[1809,90 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@111.25 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1809,90 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1809,90 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1809,90 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.26 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1809,91 INFO] (SDKD log:137) "Options": {
[1809,91 INFO] (SDKD log:137) "DelayMin": 1,
[1809,91 INFO] (SDKD log:137) "TimeRes": 1,
[1809,91 INFO] (SDKD log:137) "IterWait": 1,
[1809,92 INFO] (SDKD log:137) "DelayMax": 10
[1809,92 INFO] (SDKD log:137) },
[1809,92 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1809,92 INFO] (SDKD log:137) "DS": {
[1809,92 INFO] (SDKD log:137) "KSize": 32,
[1809,93 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1809,93 INFO] (SDKD log:137) "VSize": 128,
[1809,93 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1809,93 INFO] (SDKD log:137) "Continuous": true,
[1809,94 INFO] (SDKD log:137) "Repeat": "REP",
[1809,94 INFO] (SDKD log:137) "Count": 15000
[1809,94 INFO] (SDKD log:137) }
[1809,94 INFO] (SDKD log:137) }
[1809,94 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1809,95 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[1814,99 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1814,99 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.12
[1814,99 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@108.27 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1815,00 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1815,00 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1815,00 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.28 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1815,00 INFO] (SDKD log:137) "Options": {
[1815,00 INFO] (SDKD log:137) "DelayMin": 1,
[1815,01 INFO] (SDKD log:137) "TimeRes": 1,
[1815,01 INFO] (SDKD log:137) "IterWait": 1,
[1815,01 INFO] (SDKD log:137) "DelayMax": 10
[1815,01 INFO] (SDKD log:137) },
[1815,02 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[1815,02 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1815,02 INFO] (SDKD log:137) "DS": {
[1815,02 INFO] (SDKD log:137) "KSize": 32,
[1815,03 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1815,03 INFO] (SDKD log:137) "VSize": 128,
[1815,03 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1815,03 INFO] (SDKD log:137) "Continuous": true,
[1815,03 INFO] (SDKD log:137) "Repeat": "REP",
[1815,03 INFO] (SDKD log:137) "Count": 15000
[1815,04 INFO] (SDKD log:137) }
[1815,04 INFO] (SDKD log:137) }
[1815,04 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1815,04 INFO] (SDKD log:137) System.ObjectDisposedException: Доступ к ликвидированному объекту невозможен.
[1815,05 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[1815,05 INFO] (SDKD log:137) Имя объекта: "System.Net.Sockets.Socket".
[1815,05 INFO] (SDKD log:137) at System.Net.Sockets.Socket.get_RemoteEndPoint()
[1815,05 INFO] (SDKD log:137) at Couchbase.IO.Strategies.EapConnection.<>c__DisplayClass3`1.<Send>b__0(FormatMessageHandler m) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:строка 38
[1815,06 INFO] (SDKD log:137) at Common.Logging.Factory.AbstractLogger.FormatMessageCallbackFormattedMessage.ToString()
[1815,06 INFO] (SDKD log:137) at log4net.ObjectRenderer.DefaultRenderer.RenderObject(RendererMap rendererMap, Object obj, TextWriter writer)
[1815,06 INFO] (SDKD log:137) at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj, TextWriter writer)
[1815,06 INFO] (SDKD log:137) log4net:ERROR RendererMap: Exception while rendering object of type [Common.Logging.Factory.AbstractLogger+FormatMessageCallbackFormattedMessage]
[1815,07 INFO] (SDKD log:137) System.ObjectDisposedException: Доступ к ликвидированному объекту невозможен.
[1815,07 INFO] (SDKD log:137) Имя объекта: "System.Net.Sockets.Socket".
[1815,07 INFO] (SDKD log:137) at System.Net.Sockets.Socket.get_RemoteEndPoint()
[1815,07 INFO] (SDKD log:137) at Couchbase.IO.Strategies.EapConnection.<>c__DisplayClass3`1.<Send>b__0(FormatMessageHandler m) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:строка 38
[1815,07 INFO] (SDKD log:137) at Common.Logging.Factory.AbstractLogger.FormatMessageCallbackFormattedMessage.ToString()
[1815,08 INFO] (SDKD log:137) at log4net.ObjectRenderer.DefaultRenderer.RenderObject(RendererMap rendererMap, Object obj, TextWriter writer)
[1815,08 INFO] (SDKD log:137) at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj, TextWriter writer)
[1820,02 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1820,02 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.11
[1820,02 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@107.29 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1820,02 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1820,03 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1820,03 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.30 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1820,03 INFO] (SDKD log:137) "Options": {
[1820,03 INFO] (SDKD log:137) "DelayMin": 1,
[1820,03 INFO] (SDKD log:137) "TimeRes": 1,
[1820,04 INFO] (SDKD log:137) "IterWait": 1,
[1820,04 INFO] (SDKD log:137) "DelayMax": 10
[1820,04 INFO] (SDKD log:137) },
[1820,04 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1820,05 INFO] (SDKD log:137) "DS": {
[1820,05 INFO] (SDKD log:137) "KSize": 32,
[1820,05 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1820,05 INFO] (SDKD log:137) "VSize": 128,
[1820,05 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1820,06 INFO] (SDKD log:137) "Continuous": true,
[1820,06 INFO] (SDKD log:137) "Repeat": "REP",
[1820,06 INFO] (SDKD log:137) "Count": 15000
[1820,06 INFO] (SDKD log:137) }
[1820,06 INFO] (SDKD log:137) }
[1820,06 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1820,07 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[1821,53 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 23344
[1825,14 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1825,14 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.14
[1825,15 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@110.31 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1825,15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1825,15 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1825,15 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.32 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1825,16 INFO] (SDKD log:137) "Options": {
[1825,16 INFO] (SDKD log:137) "DelayMin": 1,
[1825,16 INFO] (SDKD log:137) "TimeRes": 1,
[1825,16 INFO] (SDKD log:137) "IterWait": 1,
[1825,17 INFO] (SDKD log:137) "DelayMax": 10
[1825,17 INFO] (SDKD log:137) },
[1825,17 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1825,17 INFO] (SDKD log:137) "DS": {
[1825,17 INFO] (SDKD log:137) "KSize": 32,
[1825,18 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1825,18 INFO] (SDKD log:137) "VSize": 128,
[1825,18 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1825,18 INFO] (SDKD log:137) "Continuous": true,
[1825,18 INFO] (SDKD log:137) "Repeat": "REP",
[1825,19 INFO] (SDKD log:137) "Count": 15000
[1825,19 INFO] (SDKD log:137) }
[1825,19 INFO] (SDKD log:137) }
[1825,19 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1825,19 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120
[1830,16 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1830,16 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.13
[1830,16 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@109.33 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1830,17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1830,17 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1830,17 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.34 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1830,17 INFO] (SDKD log:137) "Options": {
[1830,17 INFO] (SDKD log:137) "DelayMin": 1,
[1830,18 INFO] (SDKD log:137) "TimeRes": 1,
[1830,18 INFO] (SDKD log:137) "IterWait": 1,
[1830,18 INFO] (SDKD log:137) "DelayMax": 10
[1830,18 INFO] (SDKD log:137) },
[1830,18 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1830,19 INFO] (SDKD log:137) "DS": {
[1830,19 INFO] (SDKD log:137) "KSize": 32,
[1830,19 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1830,19 INFO] (SDKD log:137) "VSize": 128,
[1830,20 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1830,20 INFO] (SDKD log:137) "Continuous": true,
[1830,20 INFO] (SDKD log:137) "Repeat": "REP",
[1830,20 INFO] (SDKD log:137) "Count": 15000
[1830,20 INFO] (SDKD log:137) }
[1830,21 INFO] (SDKD log:137) }
[1830,21 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1830,21 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121
[1836,39 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1836,39 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.10
[1836,40 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@106.35 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1836,40 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1836,40 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1836,40 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@122.36 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1836,41 INFO] (SDKD log:137) "Options": {
[1836,41 INFO] (SDKD log:137) "DelayMin": 1,
[1836,41 INFO] (SDKD log:137) "TimeRes": 1,
[1836,41 INFO] (SDKD log:137) "IterWait": 1,
[1836,42 INFO] (SDKD log:137) "DelayMax": 10
[1836,42 INFO] (SDKD log:137) },
[1836,42 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1836,42 INFO] (SDKD log:137) "DS": {
[1836,43 INFO] (SDKD log:137) "KSize": 32,
[1836,43 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1836,43 INFO] (SDKD log:137) "VSize": 128,
[1836,43 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1836,44 INFO] (SDKD log:137) "Continuous": true,
[1836,44 INFO] (SDKD log:137) "Repeat": "REP",
[1836,44 INFO] (SDKD log:137) "Count": 15000
[1836,44 INFO] (SDKD log:137) }
[1836,44 INFO] (SDKD log:137) }
[1836,45 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1836,45 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 122
[1841,55 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1841,55 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@113.18
[1841,55 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@113.37 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1841,56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1841,56 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1841,56 INFO] (SDKD log:137) "Options": {
[1841,56 INFO] (SDKD log:137) "DelayMin": 1,
[1841,57 INFO] (SDKD log:137) "TimeRes": 1,
[1841,57 INFO] (SDKD log:137) "IterWait": 1,
[1841,57 INFO] (SDKD log:137) "DelayMax": 10
[1841,57 INFO] (SDKD log:137) },
[1841,58 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1841,58 INFO] (SDKD log:137) "DS": {
[1841,58 INFO] (SDKD log:137) "KSize": 32,
[1841,58 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1841,58 INFO] (SDKD log:137) "VSize": 128,
[1841,59 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1841,59 INFO] (SDKD log:137) "Continuous": true,
[1841,59 INFO] (SDKD log:137) "Repeat": "REP",
[1841,59 INFO] (SDKD log:137) "Count": 15000
[1841,59 INFO] (SDKD log:137) }
[1841,60 INFO] (SDKD log:137) }
[1846,71 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1846,71 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@114.20
[1846,71 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@114.38 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1846,72 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1846,72 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1846,72 INFO] (SDKD log:137) "Options": {
[1846,73 INFO] (SDKD log:137) "DelayMin": 1,
[1846,73 INFO] (SDKD log:137) "TimeRes": 1,
[1846,73 INFO] (SDKD log:137) "IterWait": 1,
[1846,73 INFO] (SDKD log:137) "DelayMax": 10
[1846,73 INFO] (SDKD log:137) },
[1846,74 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1846,74 INFO] (SDKD log:137) "DS": {
[1846,74 INFO] (SDKD log:137) "KSize": 32,
[1846,74 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1846,74 INFO] (SDKD log:137) "VSize": 128,
[1846,75 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1846,75 INFO] (SDKD log:137) "Continuous": true,
[1846,75 INFO] (SDKD log:137) "Repeat": "REP",
[1846,75 INFO] (SDKD log:137) "Count": 15000
[1846,75 INFO] (SDKD log:137) }
[1846,76 INFO] (SDKD log:137) }
[1851,69 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[1851,69 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@115.22
[1851,69 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@115.39 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1851,70 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1851,70 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1851,70 INFO] (SDKD log:137) "Options": {
[1851,71 INFO] (SDKD log:137) "DelayMin": 1,
[1851,71 INFO] (SDKD log:137) "TimeRes": 1,
[1851,71 INFO] (SDKD log:137) "IterWait": 1,
[1851,71 INFO] (SDKD log:137) "DelayMax": 10
[1851,71 INFO] (SDKD log:137) },
[1851,72 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1851,72 INFO] (SDKD log:137) "DS": {
[1851,72 INFO] (SDKD log:137) "KSize": 32,
[1851,72 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1851,72 INFO] (SDKD log:137) "VSize": 128,
[1851,73 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1851,73 INFO] (SDKD log:137) "Continuous": true,
[1851,73 INFO] (SDKD log:137) "Repeat": "REP",
[1851,73 INFO] (SDKD log:137) "Count": 15000
[1851,73 INFO] (SDKD log:137) }
[1851,74 INFO] (SDKD log:137) }
[1854,01 INFO] (SDKD log:137)
[1854,01 INFO] (SDKD log:137) unhandled Exception: System.ObjectDisposedException: Cannot access a disposed object
[1854,01 INFO] (SDKD log:137) Object name: "Object ThreadLocal has been dispose.".
[1854,01 INFO] (SDKD log:137) at System.Threading.ThreadLocal`1.GetValueSlow()
[1854,01 INFO] (SDKD log:137) at System.Threading.ThreadLocal`1.get_Value()
[1854,02 INFO] (SDKD log:137) at System.Collections.Concurrent.ConcurrentBag`1.GetThreadList(Boolean forceCreate)
[1854,02 INFO] (SDKD log:137) at System.Collections.Concurrent.ConcurrentBag`1.TryTakeOrPeek(T& result, Boolean take)
[1854,02 INFO] (SDKD log:137) at System.Collections.Concurrent.ConcurrentBag`1.TryTake(T& result)
[1854,02 INFO] (SDKD log:137) at Couchbase.IO.ConnectionPool`1.Dispose(Boolean disposing) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\ConnectionPool.cs:строка 162
[1854,02 INFO] (SDKD log:137) at Couchbase.IO.ConnectionPool`1.Dispose() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\ConnectionPool.cs:строка 147
[1854,03 INFO] (SDKD log:137) at Couchbase.IO.Strategies.DefaultIOStrategy.Dispose(Boolean disposing) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 113
[1854,03 INFO] (SDKD log:137) at Couchbase.IO.Strategies.DefaultIOStrategy.Finalize() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 122
[2244,19 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_11]
        at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_11]
[2244,22 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.40
[2244,22 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]


client logs attached

 Comments   
Comment by Jeff Morris [ 19/Aug/14 ]
Try running again with what is on github. We pushed some commits through code review today. Attach the log and also the SDKQE code that's doing the testing.

Thanks,

Jeff
Comment by Andrei Baranouski [ 27/Aug/14 ]
fixed/verified




[NCBC-607] [ssl] NullReferenceException at Couchbase.IO.Strategies.SslConnection.Send ( first CRUD operation) Created: 27/Aug/14  Updated: 02/Sep/14  Resolved: 27/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File log.txt     PNG File NCBC-607.png     File test.logs    

 Description   
call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
ig.ini -I sdkd.args --variants HYBRID -d all:trace
[0.32 INFO] (BRun run:436) Initializing S3
[1.19 INFO] (BRun run:441) Initializing history database
[1.82 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools

============================================================
Running ReAdd2-HYBRID: 2014-08-26 22:52:11
    workload=HybridWorkloadGroup
    failover/next_action=FO_READD_REBALANCE
    failover/count=2
    failover/next_delay=45
    testcase=FailoverScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1174\ReAdd2-HYBRID\08-26-14\0211
31\68eb149dd819e0fcf974500c9b7554ef

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --workload HybridWorkloadGroup --failover-next_action
 \ FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
 \ --testcase FailoverScenario -C share\rexec --rexec_path
 \ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase
        --cluster_useSSL=True
    ---------------------------------------->8


[57.94 INFO] (Workload setupDesign:63) Creating design test_design
[57.95 TRACE] (CouchbaseAdmin getResponseJson:167) PUT http://10.3.121.134:8092/default/_design/test_design
[58.19 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8092/default/_design/test_design/_view/test_view?limit=
1
[58.23 INFO] (Workload setupDesign:80) Design creation done
[58.23 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[58.24 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=true, Username=default,
Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[58.24 TRACE] (Handle sendMessageAsync:185) > {"Command":"NEWHANDLE","Handle":101,"ReqID":1,"CommandData":{"Port":8091,"Bucket":"d
efault","Options":{"SSL":true,"Username":"default","Password":"","OtherNodes":[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.
3.206",8091]]},"Hostname":"10.3.121.134"}}
[58.25 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[58.27 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[58.27 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToke
n=af08829b84f0328e
[58.30 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[58.30 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d42611
5821
[58.58 INFO] (SDKD log:137)
[58.59 INFO] (SDKD log:137) Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an obje
ct.
[198.47 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[198.48 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.2
[198.48 INFO] (SDKD log:137) at Couchbase.IO.Strategies.SslConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.
net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\SslConnection.cs:line 53
   at Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation, IConnection connection) in c:\jenkins\workspace
\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 35
   at Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-and
rei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 45
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String p
assword) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\Carr
ierPublication\CarrierPublicationProvider.cs:line 41
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-a
ndrei\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 159
   at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei
\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 84
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs
:line 134
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boo
lean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean pre
serveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[198.49 TRACE] (Handle sendMessageAsync:185) > {"Command":"CLOSEHANDLE","Handle":101,"ReqID":2,"CommandData":{}}
[198.50 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]
[199.41 INFO] (BRun uploadFile:141) Uploaded RAW => http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1174/Re
Add2-HYBRID/08-26-14/021131/68eb149dd819e0fcf974500c9b7554ef.zip
[199.42 ERROR] (BRun main:513) Caught exception
com.couchbase.sdkdclient.context.HarnessException: java.net.SocketException: Connection reset
        at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:80) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:169) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:165) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
        at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[na:1.6.0_27]


seems like it's a regression for ssl

 Comments   
Comment by Jeff Morris [ 27/Aug/14 ]
Indeed a regression bug. http://review.couchbase.org/#/c/40982/




[NCBC-620] compilation error CS2001: Source file 'Fakes\FauxBucket.cs' could not be found Created: 31/Aug/14  Updated: 02/Sep/14  Resolved: 31/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
use Change-Id: Ic9532583efe97ac7f2202d8452763e3e4a876356

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/84/console

5>CSC : error CS2001: Source file 'Fakes\FauxBucket.cs' could not be found [c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase.Tests\Couchbase.Tests.csproj]

 Comments   
Comment by Jeff Morris [ 31/Aug/14 ]
http://review.couchbase.org/#/c/41127/




[NCBC-610] [ssl] data load is very slow: 1 ops/sec(regression) Created: 28/Aug/14  Updated: 02/Sep/14  Resolved: 28/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive log_If457e837f.zip     Text File log.txt     PNG File without_ssl.png     PNG File with_ssl_If457e837fdda.png     PNG File with_ssl.png    

 Description   
Hi Jeff,

I see a significant drop in performance with ssl

please see screenshots




 Comments   
Comment by Jeff Morris [ 28/Aug/14 ]
Running against my local cluster (VMs) the performance is nearly identical. From the logs you provided, there is something not right with your config:

2014-08-27 23:35:14,921 [4] DEBUG Couchbase.Core.ClusterManager - Trying to boostrap with Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.
2014-08-27 23:35:14,939 [4] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Getting config for bucket default
2014-08-27 23:35:14,950 [4] DEBUG Couchbase.Core.ClusterManager - Creating DefaultIOStrategy
2014-08-27 23:35:14,951 [4] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Creating DefaultIOStrategy 642ecc71-964b-42bb-aff6-4d691605c598
2014-08-27 23:35:14,960 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-08-27 23:35:14,998 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: 2fee4eef-d170-4af2-b92f-4856dfc8e359 | 10.3.121.134:11207 | [0, 0] - 154ff9f2-fc7c-452a-98f7-614407abd6dc - Disposed: False
2014-08-27 23:35:17,517 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-08-27 23:35:17,529 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: 25f51e8c-8149-4415-82a3-16c8bd183139 | 10.3.121.134:11207 | [0, 1] - 154ff9f2-fc7c-452a-98f7-614407abd6dc - Disposed: False
2014-08-27 23:35:20,035 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 25f51e8c-8149-4415-82a3-16c8bd183139 on 10.3.121.134:11207 - 154ff9f2-fc7c-452a-98f7-614407abd6dc
2014-08-27 23:35:20,038 [4] WARN Couchbase.Core.ClusterManager - Couchbase.Configuration.ConfigException: Could not retrieve configuration for default. Reason: ??
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 67
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 159

After trying CCCP, the client defaults to HTTP streaming and configuration continues. But every op fails:

2014-08-27 23:35:33,050 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: f268e1c1-1cbf-41bb-ba70-3c88ae9f4e14 on 10.3.121.135:11207 - 6b53a72a-c726-4641-b34c-f30f71ba14bd
2014-08-27 23:35:33,052 [4] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation doesn't support retries for key ViewFillerSeedrep0rep0rep0
2014-08-27 23:35:33,052 [4] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key ViewFillerSeedrep0rep0rep0 failed after 0 retries. Reason: ?
2014-08-27 23:35:33,065 [4] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#429 on thread 4
2014-08-27 23:35:33,065 [4] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 58 for key ViewFillerSeedrep1rep1rep1 - rev0
2014-08-27 23:35:33,065 [4] DEBUG Couchbase.Core.Server - Sending ViewFillerSeedrep1rep1rep1 using server 10.3.121.134:11207
2014-08-27 23:35:33,066 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-08-27 23:35:33,079 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: 471d6ca5-cd63-48da-8d99-369897e727b2 | 10.3.121.134:11207 | [0, 1] - 9884859d-0bcd-4732-

Try checking your configuration and see if their is some error there.

-Jeff
Comment by Andrei Baranouski [ 28/Aug/14 ]
Hi Jeff,

I I rolled back couchbase-net-client to version

$ git reset --hard HEAD^
HEAD is now at 0b95eb0 NCBC-604: NullReferenceException in OperationBase`1.HandleClientError

Administrator@WIN-335SPEPOCGT /c/jenkins/workspace/.net-sdkdclient-ng2-andrei/couchbase-net-client (master)
$ git log
WARNING: terminal is not fully functional
commit 0b95eb0e888c394c00048b66f15fe7fb0fbf5146
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Tue Aug 26 14:28:57 2014 -0700

    NCBC-604: NullReferenceException in OperationBase`1.HandleClientError

    Change-Id: If457e837fdda073b51b370a588863200910a29be
    Reviewed-on: http://review.couchbase.org/40938
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

and loading looks fine( there are no changes in sdkd frameworks)

see attached screenshot and log with this revision



Comment by Jeff Morris [ 28/Aug/14 ]
Thanks Andrei, I'll look deeper into it this afternoon, as you mentioned it's probably a regression.
Comment by Jeff Morris [ 28/Aug/14 ]
http://review.couchbase.org/#/c/41064/
Comment by Andrei Baranouski [ 30/Aug/14 ]
yeah, it solves the problem




[NCBC-597] [vpn run] Couchbase.Configuration.Server.Serialization.BootstrapException: Could not bootstrap from configured servers list Created: 21/Aug/14  Updated: 30/Sep/14  Resolved: 25/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-597.log.zip    
Issue Links:
Relates to
relates to NCBC-600 Update Client Client Configuration Wh... Resolved

 Description   
git log
commit 86219ee99ccafdd00c98d577c9bc2b97efef35bb
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Aug 20 17:44:45 2014 -0700

    NCBC-592: AuthenticationException in CouchbaseBucket.SendWithRetry

c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,17 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/count=2
    workload=HybridWorkloadGroup
    rebalance/mode=out
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1173\Rb2Out-HYBRID\08-21-14\018604\baf8737ee95c04b1942a18c39e286fd1

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-count 2 --workload HybridWorkloadGroup
 \ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
 \ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1,50 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1,77 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,78 INFO] (RunContext run:124) Starting cluster and driver
[1,79 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,79 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1,79 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2,13 INFO] (SDKD log:137) --- Logging Self-Test ---
[2,14 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[2,14 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[2,14 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[2,14 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[2,18 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,32 INFO] (SDKD log:137) Call SweepOnce()
[2,32 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,32 INFO] (SDKD log:137) Call SweepOnce()
[2,32 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,32 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[2,43 INFO] (SDKD log:137) Call SweepOnce()
[2,43 INFO] (SDKD log:137) Call SweepOnce()
[2,44 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[13,36 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.145:8091,ns_1@10.3.4.145>
[13,83 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.146:8091,ns_1@10.3.4.146>
[14,26 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[14,72 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[14,72 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[14,72 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[14,73 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Andrei\.ssh\id_rsa
[16,79 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[16,79 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[16,81 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[16,83 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[16,83 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[16,83 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[16,97 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[16,98 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[17,54 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@241a6856
[17,54 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@412effef
[17,55 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1deaf20
[17,99 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@7a2f1c2a
[17,99 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@7c737a1a
[35,10 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[41,18 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[44,39 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[56,06 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[56,91 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[56,91 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[57,12 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[57,34 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[66,15 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[68,68 INFO] (RunContext run:143) Driver and cluster initialized
[68,92 INFO] (RunContext call:167) Running scenario..
[68,93 INFO] (Scenario run:72) Starting RAMP phase
[68,93 INFO] (Workload setupDesign:63) Creating design test_design
[70,20 INFO] (Workload setupDesign:80) Design creation done
[70,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[70,20 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[70,20 INFO] (SDKD log:137) Call SweepOnce()
[70,20 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[70,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[70,29 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[70,29 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[70,31 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[70,31 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[78,64 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[78,64 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {Options={DelayMin=0, TimeRes=0, IterWait=1, DelayMax=0}, Schema={InflateLevel=40, InflateContent=meh}, DSType=DSTYPE_SEEDED, DS={KSize=12, KSeed=ViewFillerSeed, VSize=512, VSeed=Vie
wFillerSeed, Continuous=false, Repeat=rep, Count=1000}}
[78,65 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[78,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[78,66 INFO] (SDKD log:137) "Options": {
[78,66 INFO] (SDKD log:137) "DelayMin": 0,
[78,66 INFO] (SDKD log:137) "TimeRes": 0,
[78,67 INFO] (SDKD log:137) "IterWait": 1,
[78,67 INFO] (SDKD log:137) "DelayMax": 0
[78,67 INFO] (SDKD log:137) },
[78,67 INFO] (SDKD log:137) "Schema": {
[78,67 INFO] (SDKD log:137) "InflateLevel": 40,
[78,68 INFO] (SDKD log:137) "InflateContent": "meh"
[78,68 INFO] (SDKD log:137) },
[78,68 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[78,69 INFO] (SDKD log:137) "DS": {
[78,69 INFO] (SDKD log:137) "KSize": 12,
[78,69 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[78,69 INFO] (SDKD log:137) "VSize": 512,
[78,70 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[78,70 INFO] (SDKD log:137) "Continuous": false,
[78,70 INFO] (SDKD log:137) "Repeat": "rep",
[78,71 INFO] (SDKD log:137) "Count": 1000
[78,71 INFO] (SDKD log:137) }
[78,71 INFO] (SDKD log:137) }
[290,18 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[290,18 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[290,19 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[290,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[290,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[290,19 INFO] (Workload call:112) Preloading items into cluster before workload
[290,19 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 2092826221
[290,20 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[290,20 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()2092826221
[290,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[290,20 INFO] (SDKD log:137) Call SweepOnce()
[290,21 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[290,21 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[290,22 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[296,87 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[296,88 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@102.5 => {Options={DelayMin=0, TimeRes=0, IterWait=100, DelayMax=0}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=false, Repeat=REP, Coun
t=15000}}
[296,88 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[296,88 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[296,88 INFO] (SDKD log:137) "Options": {
[296,89 INFO] (SDKD log:137) "DelayMin": 0,
[296,89 INFO] (SDKD log:137) "TimeRes": 0,
[296,89 INFO] (SDKD log:137) "IterWait": 100,
[296,89 INFO] (SDKD log:137) "DelayMax": 0
[296,90 INFO] (SDKD log:137) },
[296,90 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[296,90 INFO] (SDKD log:137) "DS": {
[296,90 INFO] (SDKD log:137) "KSize": 32,
[296,90 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[296,91 INFO] (SDKD log:137) "VSize": 128,
[296,91 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[296,91 INFO] (SDKD log:137) "Continuous": false,
[296,91 INFO] (SDKD log:137) "Repeat": "REP",
[296,92 INFO] (SDKD log:137) "Count": 15000
[296,92 INFO] (SDKD log:137) }
[296,92 INFO] (SDKD log:137) }
[347,55 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[1868,44 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@102.5 => {[OK]: 14941, [MEMD:GENERIC]: 59}
[1868,44 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[1868,44 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[1868,44 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[1868,44 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 1781340248
[1868,45 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.11 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,45 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.10 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,45 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,47 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.9 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.8 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.15 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.16 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.14 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.13 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()1781340248
[1868,48 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.12 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1868,48 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[1868,49 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,49 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,49 INFO] (SDKD log:137) Call SweepOnce()
[1868,49 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,50 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,50 INFO] (SDKD log:137) Call SweepOnce()
[1868,50 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,50 INFO] (SDKD log:137) Call SweepOnce()
[1868,51 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,51 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,51 INFO] (SDKD log:137) Call SweepOnce()
[1868,51 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,52 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,52 INFO] (SDKD log:137) Call SweepOnce()
[1868,52 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,53 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,53 INFO] (SDKD log:137) Call SweepOnce()
[1868,54 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,54 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,54 INFO] (SDKD log:137) Call SweepOnce()
[1868,55 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,55 INFO] (SDKD log:137) Call SweepOnce()
[1868,55 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,56 INFO] (SDKD log:137) Call SweepOnce()
[1868,56 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,56 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1868,56 INFO] (SDKD log:137) Call SweepOnce()
[1868,57 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1868,57 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[1868,57 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[1868,57 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[1868,58 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[1868,58 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[1868,58 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112
[1868,58 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109
[1868,58 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111
[1868,59 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[1868,59 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110
[1874,52 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.11
[1874,62 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@107.17 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1874,62 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1874,62 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.18 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1874,63 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1874,63 INFO] (SDKD log:137) "Options": {
[1874,63 INFO] (SDKD log:137) "DelayMin": 1,
[1874,63 INFO] (SDKD log:137) "TimeRes": 1,
[1874,64 INFO] (SDKD log:137) "IterWait": 1,
[1874,64 INFO] (SDKD log:137) "DelayMax": 10
[1874,64 INFO] (SDKD log:137) },
[1874,64 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1874,65 INFO] (SDKD log:137) "DS": {
[1874,65 INFO] (SDKD log:137) "KSize": 32,
[1874,65 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1874,66 INFO] (SDKD log:137) "VSize": 128,
[1874,66 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1874,66 INFO] (SDKD log:137) "Continuous": true,
[1874,67 INFO] (SDKD log:137) "Repeat": "REP",
[1874,67 INFO] (SDKD log:137) "Count": 15000
[1874,67 INFO] (SDKD log:137) }
[1874,67 INFO] (SDKD log:137) }
[1874,67 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1874,68 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1874,68 INFO] (SDKD log:137) Call SweepOnce()
[1874,68 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113
[1880,35 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.10
[1880,35 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@106.19 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1880,36 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1880,36 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1880,36 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.20 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1880,37 INFO] (SDKD log:137) "Options": {
[1880,37 INFO] (SDKD log:137) "DelayMin": 1,
[1880,38 INFO] (SDKD log:137) "TimeRes": 1,
[1880,38 INFO] (SDKD log:137) "IterWait": 1,
[1880,38 INFO] (SDKD log:137) "DelayMax": 10
[1880,38 INFO] (SDKD log:137) },
[1880,39 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1880,39 INFO] (SDKD log:137) "DS": {
[1880,39 INFO] (SDKD log:137) "KSize": 32,
[1880,39 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1880,40 INFO] (SDKD log:137) "VSize": 128,
[1880,40 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1880,40 INFO] (SDKD log:137) "Continuous": true,
[1880,41 INFO] (SDKD log:137) "Repeat": "REP",
[1880,41 INFO] (SDKD log:137) "Count": 15000
[1880,41 INFO] (SDKD log:137) }
[1880,41 INFO] (SDKD log:137) }
[1880,42 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1880,42 INFO] (SDKD log:137) Call SweepOnce()
[1880,42 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1880,42 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114
[1886,61 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[1886,61 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@103.21 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1886,61 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1886,62 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1886,62 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.22 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1886,62 INFO] (SDKD log:137) "Options": {
[1886,63 INFO] (SDKD log:137) "DelayMin": 1,
[1886,63 INFO] (SDKD log:137) "TimeRes": 1,
[1886,63 INFO] (SDKD log:137) "IterWait": 1,
[1886,64 INFO] (SDKD log:137) "DelayMax": 10
[1886,64 INFO] (SDKD log:137) },
[1886,64 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1886,65 INFO] (SDKD log:137) "DS": {
[1886,65 INFO] (SDKD log:137) "KSize": 32,
[1886,65 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1886,66 INFO] (SDKD log:137) "VSize": 128,
[1886,66 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1886,66 INFO] (SDKD log:137) "Continuous": true,
[1886,67 INFO] (SDKD log:137) "Repeat": "REP",
[1886,67 INFO] (SDKD log:137) "Count": 15000
[1886,68 INFO] (SDKD log:137) }
[1886,68 INFO] (SDKD log:137) }
[1886,68 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1886,68 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1886,68 INFO] (SDKD log:137) Call SweepOnce()
[1886,69 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115
[1892,19 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.8
[1892,19 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@104.23 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1892,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1892,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1892,20 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.24 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1892,20 INFO] (SDKD log:137) "Options": {
[1892,20 INFO] (SDKD log:137) "DelayMin": 1,
[1892,21 INFO] (SDKD log:137) "TimeRes": 1,
[1892,21 INFO] (SDKD log:137) "IterWait": 1,
[1892,21 INFO] (SDKD log:137) "DelayMax": 10
[1892,21 INFO] (SDKD log:137) },
[1892,22 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1892,22 INFO] (SDKD log:137) "DS": {
[1892,22 INFO] (SDKD log:137) "KSize": 32,
[1892,22 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1892,23 INFO] (SDKD log:137) "VSize": 128,
[1892,23 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1892,23 INFO] (SDKD log:137) "Continuous": true,
[1892,24 INFO] (SDKD log:137) "Repeat": "REP",
[1892,24 INFO] (SDKD log:137) "Count": 15000
[1892,24 INFO] (SDKD log:137) }
[1892,24 INFO] (SDKD log:137) }
[1892,25 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1892,25 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1892,25 INFO] (SDKD log:137) Call SweepOnce()
[1892,25 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116
[1897,18 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.9
[1897,18 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@105.25 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1897,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1897,19 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1897,19 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.26 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1897,19 INFO] (SDKD log:137) "Options": {
[1897,19 INFO] (SDKD log:137) "DelayMin": 1,
[1897,20 INFO] (SDKD log:137) "TimeRes": 1,
[1897,20 INFO] (SDKD log:137) "IterWait": 1,
[1897,20 INFO] (SDKD log:137) "DelayMax": 10
[1897,21 INFO] (SDKD log:137) },
[1897,21 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1897,21 INFO] (SDKD log:137) "DS": {
[1897,21 INFO] (SDKD log:137) "KSize": 32,
[1897,22 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1897,22 INFO] (SDKD log:137) "VSize": 128,
[1897,22 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1897,22 INFO] (SDKD log:137) "Continuous": true,
[1897,23 INFO] (SDKD log:137) "Repeat": "REP",
[1897,23 INFO] (SDKD log:137) "Count": 15000
[1897,23 INFO] (SDKD log:137) }
[1897,24 INFO] (SDKD log:137) }
[1897,24 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1897,24 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1897,25 INFO] (SDKD log:137) Call SweepOnce()
[1897,25 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117
[1903,07 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.16
[1903,07 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@112.27 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1903,08 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1903,08 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1903,08 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.28 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1903,08 INFO] (SDKD log:137) "Options": {
[1903,09 INFO] (SDKD log:137) "DelayMin": 1,
[1903,09 INFO] (SDKD log:137) "TimeRes": 1,
[1903,09 INFO] (SDKD log:137) "IterWait": 1,
[1903,09 INFO] (SDKD log:137) "DelayMax": 10
[1903,10 INFO] (SDKD log:137) },
[1903,10 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1903,10 INFO] (SDKD log:137) "DS": {
[1903,10 INFO] (SDKD log:137) "KSize": 32,
[1903,11 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1903,11 INFO] (SDKD log:137) "VSize": 128,
[1903,11 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1903,12 INFO] (SDKD log:137) "Continuous": true,
[1903,12 INFO] (SDKD log:137) "Repeat": "REP",
[1903,12 INFO] (SDKD log:137) "Count": 15000
[1903,12 INFO] (SDKD log:137) }
[1903,13 INFO] (SDKD log:137) }
[1903,13 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1903,13 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1903,13 INFO] (SDKD log:137) Call SweepOnce()
[1903,14 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118
[1909,85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.13
[1909,85 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@109.29 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Coun
t=15000}}
[1909,86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[1909,86 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.30 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1909,86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1909,86 INFO] (SDKD log:137) "Options": {
[1909,87 INFO] (SDKD log:137) "DelayMin": 1,
[1909,87 INFO] (SDKD log:137) "TimeRes": 1,
[1909,87 INFO] (SDKD log:137) "IterWait": 1,
[1909,87 INFO] (SDKD log:137) "DelayMax": 10
[1909,88 INFO] (SDKD log:137) },
[1909,88 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1909,88 INFO] (SDKD log:137) "DS": {
[1909,89 INFO] (SDKD log:137) "KSize": 32,
[1909,89 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1909,89 INFO] (SDKD log:137) "VSize": 128,
[1909,89 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1909,90 INFO] (SDKD log:137) "Continuous": true,
[1909,90 INFO] (SDKD log:137) "Repeat": "REP",
[1909,91 INFO] (SDKD log:137) "Count": 15000
[1909,91 INFO] (SDKD log:137) }
[1909,91 INFO] (SDKD log:137) }
[1909,91 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1909,92 INFO] (SDKD log:137) Call SweepOnce()
[1909,92 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1909,92 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119
[1925,93 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.15
[1925,93 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@111.31 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[1925,93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[1925,93 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.32 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[1925,93 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[1925,94 INFO] (SDKD log:137) "Options": {
[1925,94 INFO] (SDKD log:137) "DelayMin": 1,
[1925,94 INFO] (SDKD log:137) "TimeRes": 1,
[1925,94 INFO] (SDKD log:137) "IterWait": 1,
[1925,95 INFO] (SDKD log:137) "DelayMax": 10
[1925,95 INFO] (SDKD log:137) },
[1925,95 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[1925,95 INFO] (SDKD log:137) "DS": {
[1925,96 INFO] (SDKD log:137) "KSize": 32,
[1925,96 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[1925,96 INFO] (SDKD log:137) "VSize": 128,
[1925,96 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[1925,97 INFO] (SDKD log:137) "Continuous": true,
[1925,97 INFO] (SDKD log:137) "Repeat": "REP",
[1925,97 INFO] (SDKD log:137) "Count": 15000
[1925,98 INFO] (SDKD log:137) }
[1925,98 INFO] (SDKD log:137) }
[1925,98 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[1925,98 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[1925,99 INFO] (SDKD log:137) Call SweepOnce()
[1925,99 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120
[2031,57 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.14
[2031,57 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@110.33 => {Options={DelayMin=1, TimeRes=1, IterWait=1, DelayMax=10}, DSType=DSTYPE_SEEDED, DS={KSize=32, KSeed=SimpleKey, VSize=128, VSeed=SimpleValue, Continuous=true, Repeat=REP, Count=15000
}}
[2031,57 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_GET
[2031,57 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[2031,57 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.34 => {Options={OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]], Username=default, SSL=false, Password=}, Bucket=default, Port=8091, Hostname=10.3.4.144}
[2031,58 INFO] (SDKD log:137) "Options": {
[2031,58 INFO] (SDKD log:137) "DelayMin": 1,
[2031,58 INFO] (SDKD log:137) "TimeRes": 1,
[2031,58 INFO] (SDKD log:137) "IterWait": 1,
[2031,58 INFO] (SDKD log:137) "DelayMax": 10
[2031,58 INFO] (SDKD log:137) },
[2031,58 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[2031,58 INFO] (SDKD log:137) "DS": {
[2031,58 INFO] (SDKD log:137) "KSize": 32,
[2031,58 INFO] (SDKD log:137) "KSeed": "SimpleKey",
[2031,59 INFO] (SDKD log:137) "VSize": 128,
[2031,59 INFO] (SDKD log:137) "VSeed": "SimpleValue",
[2031,59 INFO] (SDKD log:137) "Continuous": true,
[2031,59 INFO] (SDKD log:137) "Repeat": "REP",
[2031,59 INFO] (SDKD log:137) "Count": 15000
[2031,59 INFO] (SDKD log:137) }
[2031,59 INFO] (SDKD log:137) }
[2031,59 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2031,59 INFO] (SDKD log:137) !!!!!!!!!!!!!!!
[2031,59 INFO] (SDKD log:137) Call SweepOnce()
[2031,59 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121
[2133,15 INFO] (SDKD log:137)
[2133,15 INFO] (SDKD log:137) Unhandled Exception : Couchbase.Configuration.Server.Serialization.BootstrapException: Could not bootstrap from configured servers list.
[2133,15 INFO] (SDKD log:137) at Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig.Initialize() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\Streaming\HttpServerConfig.cs:line 52

[2133,16 INFO] (SDKD log:137) at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.StartProvider(String username, String password) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\
Streaming\HttpStreamingProvider.cs:line 238
[2133,16 INFO] (SDKD log:137) at Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.GetConfig(String bucketName, String password) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\St
reaming\HttpStreamingProvider.cs:line 53
[2133,16 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 159
[2133,16 INFO] (SDKD log:137) at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 84
[2133,16 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.Run() at c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 134
[2133,16 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[2133,17 INFO] (SDKD log:137) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[2133,17 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[2133,17 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[2133,17 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart()
[2152,56 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_11]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_11]
        at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_11]
[2152,56 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.35
[2152,56 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.8.0_11]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.8.0_11]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.8.0_11]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]

reproduce it in each run through vpn, don't see in jenkins http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/64/console

 Comments   
Comment by Jeff Morris [ 22/Aug/14 ]
What is going on here is that the while trying to bootstrap, the client has exhausted the servers available within it's client configuration. Without looking at the sdkd-net code, I am guessing it's creating a configuration with a single bootstrap server and then eventually that bootstrap server is removed from the cluster. As evidence:

Line 19: 2014-08-20 11:19:48,001 [4] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 7120: 2014-08-20 11:21:34,247 [5] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 112230: 2014-08-20 11:46:07,977 [12] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 112298: 2014-08-20 11:46:13,909 [21] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 112523: 2014-08-20 11:46:19,922 [26] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 112936: 2014-08-20 11:46:25,401 [15] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 113567: 2014-08-20 11:46:31,724 [14] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 114433: 2014-08-20 11:46:38,069 [23] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 115409: 2014-08-20 11:46:44,009 [22] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 116591: 2014-08-20 11:46:49,994 [20] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 116654: 2014-08-21 08:11:15,680 [4] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 123744: 2014-08-21 08:14:54,350 [11] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 228894: 2014-08-21 08:41:12,600 [13] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 228959: 2014-08-21 08:41:18,672 [7] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 229187: 2014-08-21 08:41:24,526 [9] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 229665: 2014-08-21 08:41:30,757 [6] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 230256: 2014-08-21 08:41:36,329 [5] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 230950: 2014-08-21 08:41:41,370 [23] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 231963: 2014-08-21 08:41:47,217 [22] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 233352: 2014-08-21 08:41:53,990 [20] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 237408: 2014-08-21 08:42:10,063 [21] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools
Line 267746: 2014-08-21 08:43:55,722 [15] INFO Couchbase.Configuration.Server.Providers.Streaming.HttpServerConfig - Bootstrapping from http://10.3.4.144:8091/pools

The bootstrap list is shuffled, but the it always tries to bootstrap off the same server.
Comment by Jeff Morris [ 22/Aug/14 ]
Another possible issue is that the client is using the HTTP provider, which means it tried the CCCP provider and failed. The CCCP provider is preferred when the server is > 2.5.
Comment by Jeff Morris [ 25/Aug/14 ]
Depending upon how the client is used, this should resolve this: http://review.couchbase.org/#/c/40850/

However, you should add more nodes to your bootstrap list. This is the list the client will shuffle through until it either finds a node to bootstrap against or cannot bootstrap (because it can't find a candidate node in _your_ provided list). For example:

 var config = new ClientConfiguration
            {
                Servers = new List<Uri>
                {
                    new Uri("http://192.168.56.101:8091/pools"),
                    new Uri("http://192.168.56.102:8091/pools"),
                    new Uri("http://192.168.56.103:8091/pools"),
                    new Uri("http://192.168.56.104:8091/pools"),
                }
            };




[NCBC-645] [ssl] test Rb2Out-HYBRID stuck with 'Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException' in client logs Created: 17/Sep/14  Updated: 30/Sep/14  Resolved: 30/Sep/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-645.zip     Text File test_logs.txt    

 Description   
c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
ig.ini -I sdkd.args --variants HYBRID -d all:debug
[3.21 INFO] (BRun run:436) Initializing S3
[7.80 INFO] (BRun run:441) Initializing history database

============================================================
Running Rb2Out-HYBRID: 2014-09-17 02:29:58
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\Rb2Out-HYBRID\09-17-14\0341
98\28852a8b5890fe94895277f14f7f4f66

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase
        --cluster_useSSL=True

[447.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[448.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[449.93 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[449.94 INFO] (Scenario run:97) CHANGE phase done
[449.95 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[539.95 INFO] (Scenario run:105) REBOUND done. Will collect results
[539.95 DEBUG] (Handle sendMessageAsync:183) > CANCEL@103.72
[539.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 103
[539.97 DEBUG] (Handle receiveMessage:158) < CANCEL@103.72
[539.97 DEBUG] (Handle sendMessageAsync:183) > CANCEL@104.72
[539.98 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 104
[2705.74 DEBUG] (Handle receiveMessage:158) < CANCEL@104.72
[2706.66 DEBUG] (Handle sendMessageAsync:183) > CANCEL@105.72
[2707.19 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 105
Terminate batch job (Y/N)? Y




part of the client logs when test stuck(02:37 - 03:14)

2014-09-17 02:37:18,228 [7] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 7
2014-09-17 02:37:18,227 [47] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 40d8fbd1-7b5d-4334-a688-065d2575cca6 on 10.3.3.206:11207 - a17e5df5-7c63-4030-ba02-41309e68ea55
2014-09-17 02:37:18,225 [21] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: abf3e988-cbfc-4795-8523-325094b9e719 on 10.3.121.134:11207 - 624b4df1-35dc-4842-8a23-7db2310b001b
2014-09-17 02:37:18,232 [50] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#808 on thread 50
2014-09-17 02:37:18,230 [9] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 9
2014-09-17 02:37:18,231 [6] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#820 on thread 6
2014-09-17 02:37:18,233 [43] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#828 on thread 43
2014-09-17 02:37:18,253 [18] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#840 on thread 18
2014-09-17 02:37:18,252 [28] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 28
2014-09-17 02:37:18,254 [53] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#798 on thread 53
2014-09-17 02:37:18,258 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 42
2014-09-17 02:37:18,258 [46] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 981 for key SimpleKeyREP5537REP5537REP5537REP5537REP5537 - rev798
2014-09-17 02:37:18,210 [49] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 515 for key SimpleKeyREP5382REP5382REP5382REP5382REP5382 - rev870
2014-09-17 02:37:19,297 [59] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [13] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [62] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [20] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [12] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [15] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,301 [61] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [60] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,300 [11] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:51,391 [7] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 427 for key SimpleKeyREP5352REP5352REP5352REP5352REP5352 - rev870
2014-09-17 03:14:51,391 [30] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 226 for key SimpleKeyREP5697REP5697REP5697REP5697REP5697 - rev870
2014-09-17 03:14:51,391 [24] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 958 for key SimpleKeyREP5964REP5964REP5964REP5964REP5964 - rev817
2014-09-17 03:14:51,391 [48] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 456 for key SimpleKeyREP5799REP5799REP5799REP5799REP5799 - rev810
2014-09-17 03:14:51,391 [8] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 461 for key SimpleKeyREP6028REP6028REP6028REP6028REP6028 - rev799
2014-09-17 03:14:51,334 [49] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5382REP5382REP5382REP5382REP5382 using server 10.3.121.134:11207
2014-09-17 03:14:51,391 [44] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 246 for key SimpleKeyREP5854REP5854REP5854REP5854REP5854 - rev870
2014-09-17 03:14:51,391 [6] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 34 for key SimpleKeyREP5909REP5909REP5909REP5909REP5909 - rev820
2014-09-17 03:14:51,391 [9] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 728 for key SimpleKeyREP5752REP5752REP5752REP5752REP5752 - rev800
2014-09-17 03:14:51,391 [43] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 554 for key SimpleKeyREP5750REP5750REP5750REP5750REP5750 - rev828
2014-09-17 03:14:51,391 [50] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 109 for key SimpleKeyREP5503REP5503REP5503REP5503REP5503 - rev808
2014-09-17 03:14:51,391 [18] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 633 for key SimpleKeyREP5670REP5670REP5670REP5670REP5670 - rev840
2014-09-17 03:14:51,391 [28] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 825 for key SimpleKeyREP5706REP5706REP5706REP5706REP5706 - rev800
2014-09-17 03:14:51,391 [53] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 964 for key SimpleKeyREP5289REP5289REP5289REP5289REP5289 - rev798
2014-09-17 03:14:51,391 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 610 for key SimpleKeyREP5850REP5850REP5850REP5850REP5850 - rev870
2014-09-17 03:14:51,334 [46] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5537REP5537REP5537REP5537REP5537 using server 10.3.3.206:11207
2014-09-17 03:14:51,631 [45] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5291REP5291REP5291REP5291REP5291
2014-09-17 03:14:51,631 [47] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5660REP5660REP5660REP5660REP5660
2014-09-17 03:14:51,631 [21] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5520REP5520REP5520REP5520REP5520
2014-09-17 03:14:51,632 [52] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5089REP5089REP5089REP5089REP5089
2014-09-17 02:37:19,300 [63] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:54,754 [7] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5352REP5352REP5352REP5352REP5352 using server 10.3.3.206:11207
2014-09-17 03:14:54,756 [49] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: c1a7c0f9-9ae6-406a-870f-221d1cd5abc7 | 10.3.121.134:11207 | [0, 2] - 45e4bde8-afd4-4084-b4fe-1952631c5f47 - Disposed: False
2014-09-17 03:14:54,760 [18] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5670REP5670REP5670REP5670REP5670 using server 10.3.3.206:11207
2014-09-17 03:14:54,760 [28] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5706REP5706REP5706REP5706REP5706 using server 10.3.3.206:11207
2014-09-17 03:14:54,762 [46] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: dfdafeeb-7f31-4104-94ea-a7c9c951aa6e | 10.3.3.206:11207 | [0, 2] - 728e6fe1-8778-4acd-b15d-832d832b0a1a - Disposed: False
2014-09-17 03:14:54,760 [53] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5289REP5289REP5289REP5289REP5289 using server 10.3.3.206:11207
2014-09-17 03:14:54,761 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5850REP5850REP5850REP5850REP5850 using server 10.3.3.206:11207
2014-09-17 03:14:54,749 [20] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,750 [15] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [59] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,747 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,748 [62] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [50] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5503REP5503REP5503REP5503REP5503 using server 10.3.121.134:11207
2014-09-17 03:14:54,750 [12] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,758 [9] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5752REP5752REP5752REP5752REP5752 using server 10.3.3.206:11207
2014-09-17 03:14:54,754 [30] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5697REP5697REP5697REP5697REP5697 using server 10.3.121.134:11207
2014-09-17 03:14:54,752 [60] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,753 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,755 [48] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5799REP5799REP5799REP5799REP5799 using server 10.3.3.206:11207



 Comments   
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you define "stuck"? Seeing exceptions in the logs is expected, but if the client has stopped completely, this is definitely a blocking issue.

-Jeff
Comment by Andrei Baranouski [ 17/Sep/14 ]
yes, the client has stopped completely
 I only got a new batch of activities (in logs), after I click Ctrl+c in terminal where I run the tests.
That's all I can tell you now. let me know if you need some detail on this issue
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you do a "Debug->Break All" and look at the threads to see where it's blocking? I am guessing it's on one of the Wait()'s that doesn't have a timeout set.

Thanks,

Jeff
Comment by Jeff Morris [ 29/Sep/14 ]
I doubt it's related and I may create an MB, but view performance is erratic (~90ms to ~15s) during a node removal/fail-over and subsequent rebalance:

Line 338: 2014-09-29 19:12:39,724 [Runner thread] DEBUG Wintellect.OperationTimer2 - 330 (GCs= 2)
Line 340: 2014-09-29 19:12:40,134 [Runner thread] DEBUG Wintellect.OperationTimer2 - 376 (GCs= 2)
Line 342: 2014-09-29 19:12:40,484 [Runner thread] DEBUG Wintellect.OperationTimer2 - 326 (GCs= 2)
Line 344: 2014-09-29 19:12:40,860 [Runner thread] DEBUG Wintellect.OperationTimer2 - 344 (GCs= 2)
Line 346: 2014-09-29 19:12:41,297 [Runner thread] DEBUG Wintellect.OperationTimer2 - 408 (GCs= 2)
Line 348: 2014-09-29 19:12:41,656 [Runner thread] DEBUG Wintellect.OperationTimer2 - 325 (GCs= 2)
Line 350: 2014-09-29 19:12:42,070 [Runner thread] DEBUG Wintellect.OperationTimer2 - 380 (GCs= 2)
Line 352: 2014-09-29 19:12:42,519 [Runner thread] DEBUG Wintellect.OperationTimer2 - 422 (GCs= 2)
Line 354: 2014-09-29 19:12:43,112 [Runner thread] DEBUG Wintellect.OperationTimer2 - 558 (GCs= 2)
Line 356: 2014-09-29 19:12:43,767 [Runner thread] DEBUG Wintellect.OperationTimer2 - 620 (GCs= 2)
Line 358: 2014-09-29 19:12:44,311 [Runner thread] DEBUG Wintellect.OperationTimer2 - 514 (GCs= 2)
Line 360: 2014-09-29 19:12:44,889 [Runner thread] DEBUG Wintellect.OperationTimer2 - 550 (GCs= 2)
Line 362: 2014-09-29 19:12:45,555 [Runner thread] DEBUG Wintellect.OperationTimer2 - 620 (GCs= 2)
Line 364: 2014-09-29 19:12:46,457 [Runner thread] DEBUG Wintellect.OperationTimer2 - 855 (GCs= 2)
Line 366: 2014-09-29 19:12:47,389 [Runner thread] DEBUG Wintellect.OperationTimer2 - 878 (GCs= 2)
Line 368: 2014-09-29 19:12:53,148 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,712 (GCs= 2)
Line 370: 2014-09-29 19:12:53,917 [Runner thread] DEBUG Wintellect.OperationTimer2 - 739 (GCs= 2)
Line 372: 2014-09-29 19:12:54,685 [Runner thread] DEBUG Wintellect.OperationTimer2 - 733 (GCs= 2)
Line 374: 2014-09-29 19:13:00,458 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,734 (GCs= 2)
Line 376: 2014-09-29 19:13:01,153 [Runner thread] DEBUG Wintellect.OperationTimer2 - 665 (GCs= 2)
Line 378: 2014-09-29 19:13:01,805 [Runner thread] DEBUG Wintellect.OperationTimer2 - 627 (GCs= 2)
Line 380: 2014-09-29 19:13:02,406 [Runner thread] DEBUG Wintellect.OperationTimer2 - 559 (GCs= 2)
Line 382: 2014-09-29 19:13:02,834 [Runner thread] DEBUG Wintellect.OperationTimer2 - 396 (GCs= 2)
Line 384: 2014-09-29 19:13:03,476 [Runner thread] DEBUG Wintellect.OperationTimer2 - 615 (GCs= 2)
Line 386: 2014-09-29 19:13:14,271 [Runner thread] DEBUG Wintellect.OperationTimer2 - 10,763 (GCs= 2)
Line 388: 2014-09-29 19:13:14,995 [Runner thread] DEBUG Wintellect.OperationTimer2 - 693 (GCs= 2)
Line 390: 2014-09-29 19:13:20,586 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,559 (GCs= 3)
Line 392: 2014-09-29 19:13:21,353 [Runner thread] DEBUG Wintellect.OperationTimer2 - 731 (GCs= 2)
Line 394: 2014-09-29 19:13:21,933 [Runner thread] DEBUG Wintellect.OperationTimer2 - 550 (GCs= 2)
Line 396: 2014-09-29 19:13:22,508 [Runner thread] DEBUG Wintellect.OperationTimer2 - 542 (GCs= 2)
Line 398: 2014-09-29 19:13:23,228 [Runner thread] DEBUG Wintellect.OperationTimer2 - 689 (GCs= 2)
Line 400: 2014-09-29 19:13:23,739 [Runner thread] DEBUG Wintellect.OperationTimer2 - 477 (GCs= 2)
Line 402: 2014-09-29 19:13:29,418 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,649 (GCs= 3)
Line 404: 2014-09-29 19:13:30,048 [Runner thread] DEBUG Wintellect.OperationTimer2 - 590 (GCs= 2)
Line 406: 2014-09-29 19:13:30,639 [Runner thread] DEBUG Wintellect.OperationTimer2 - 557 (GCs= 2)
Line 408: 2014-09-29 19:13:31,193 [Runner thread] DEBUG Wintellect.OperationTimer2 - 524 (GCs= 2)
Line 410: 2014-09-29 19:13:31,825 [Runner thread] DEBUG Wintellect.OperationTimer2 - 599 (GCs= 2)
Line 412: 2014-09-29 19:13:32,429 [Runner thread] DEBUG Wintellect.OperationTimer2 - 573 (GCs= 2)
Line 414: 2014-09-29 19:13:48,187 [Runner thread] DEBUG Wintellect.OperationTimer2 - 15,732 (GCs= 3)
Line 416: 2014-09-29 19:13:48,806 [Runner thread] DEBUG Wintellect.OperationTimer2 - 578 (GCs= 2)
Line 418: 2014-09-29 19:13:49,433 [Runner thread] DEBUG Wintellect.OperationTimer2 - 591 (GCs= 2)
Line 420: 2014-09-29 19:13:50,060 [Runner thread] DEBUG Wintellect.OperationTimer2 - 594 (GCs= 2)
Line 422: 2014-09-29 19:13:50,912 [Runner thread] DEBUG Wintellect.OperationTimer2 - 807 (GCs= 2)
Line 424: 2014-09-29 19:13:51,553 [Runner thread] DEBUG Wintellect.OperationTimer2 - 609 (GCs= 2)
Line 426: 2014-09-29 19:13:52,447 [Runner thread] DEBUG Wintellect.OperationTimer2 - 868 (GCs= 2)
Line 428: 2014-09-29 19:13:53,022 [Runner thread] DEBUG Wintellect.OperationTimer2 - 541 (GCs= 2)
Line 430: 2014-09-29 19:13:53,711 [Runner thread] DEBUG Wintellect.OperationTimer2 - 649 (GCs= 2)
Line 432: 2014-09-29 19:13:59,399 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,655 (GCs= 3)
Line 434: 2014-09-29 19:14:00,317 [Runner thread] DEBUG Wintellect.OperationTimer2 - 882 (GCs= 2)
Line 436: 2014-09-29 19:14:00,949 [Runner thread] DEBUG Wintellect.OperationTimer2 - 600 (GCs= 2)
Line 438: 2014-09-29 19:14:01,647 [Runner thread] DEBUG Wintellect.OperationTimer2 - 669 (GCs= 2)
Line 440: 2014-09-29 19:14:02,398 [Runner thread] DEBUG Wintellect.OperationTimer2 - 712 (GCs= 2)
Line 442: 2014-09-29 19:14:03,046 [Runner thread] DEBUG Wintellect.OperationTimer2 - 616 (GCs= 2)
Line 444: 2014-09-29 19:14:03,539 [Runner thread] DEBUG Wintellect.OperationTimer2 - 462 (GCs= 2)
Line 446: 2014-09-29 19:14:03,965 [Runner thread] DEBUG Wintellect.OperationTimer2 - 395 (GCs= 2)
Line 448: 2014-09-29 19:14:04,393 [Runner thread] DEBUG Wintellect.OperationTimer2 - 395 (GCs= 2)
Line 450: 2014-09-29 19:14:04,980 [Runner thread] DEBUG Wintellect.OperationTimer2 - 562 (GCs= 2)
Line 452: 2014-09-29 19:14:10,189 [Runner thread] DEBUG Wintellect.OperationTimer2 - 5,168 (GCs= 2)
Line 454: 2014-09-29 19:14:10,368 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 456: 2014-09-29 19:14:10,555 [Runner thread] DEBUG Wintellect.OperationTimer2 - 162 (GCs= 2)
Line 458: 2014-09-29 19:14:10,714 [Runner thread] DEBUG Wintellect.OperationTimer2 - 122 (GCs= 2)
Line 460: 2014-09-29 19:14:10,920 [Runner thread] DEBUG Wintellect.OperationTimer2 - 175 (GCs= 2)
Line 462: 2014-09-29 19:14:11,088 [Runner thread] DEBUG Wintellect.OperationTimer2 - 142 (GCs= 2)
Line 464: 2014-09-29 19:14:11,240 [Runner thread] DEBUG Wintellect.OperationTimer2 - 120 (GCs= 2)
Line 466: 2014-09-29 19:14:11,443 [Runner thread] DEBUG Wintellect.OperationTimer2 - 174 (GCs= 2)
Line 468: 2014-09-29 19:14:11,620 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 470: 2014-09-29 19:14:11,805 [Runner thread] DEBUG Wintellect.OperationTimer2 - 152 (GCs= 2)
Line 472: 2014-09-29 19:14:12,008 [Runner thread] DEBUG Wintellect.OperationTimer2 - 174 (GCs= 2)
Line 474: 2014-09-29 19:14:12,190 [Runner thread] DEBUG Wintellect.OperationTimer2 - 158 (GCs= 2)
Line 476: 2014-09-29 19:14:12,353 [Runner thread] DEBUG Wintellect.OperationTimer2 - 132 (GCs= 2)
Line 478: 2014-09-29 19:14:12,530 [Runner thread] DEBUG Wintellect.OperationTimer2 - 149 (GCs= 2)
Line 480: 2014-09-29 19:14:12,724 [Runner thread] DEBUG Wintellect.OperationTimer2 - 171 (GCs= 2)
Line 482: 2014-09-29 19:14:12,883 [Runner thread] DEBUG Wintellect.OperationTimer2 - 126 (GCs= 2)
Line 484: 2014-09-29 19:14:13,053 [Runner thread] DEBUG Wintellect.OperationTimer2 - 136 (GCs= 2)
Line 486: 2014-09-29 19:14:13,221 [Runner thread] DEBUG Wintellect.OperationTimer2 - 145 (GCs= 2)
Line 488: 2014-09-29 19:14:13,349 [Runner thread] DEBUG Wintellect.OperationTimer2 - 96 (GCs= 2)

Note that all of those requests were against the node that was not removed from the cluster. They were also the same view query.
Comment by Andrei Baranouski [ 30/Sep/14 ]
can't reproduce with commit https://github.com/couchbase/couchbase-net-client/commit/747fa5523cf0b8cbefb9f615b877c41136b4fec7




[NCBC-649] infinite loop 'Trying again' with 'No connections currently available' for ReAdd2 test when nodes returned to cluster Created: 19/Sep/14  Updated: 30/Sep/14  Resolved: 30/Sep/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-649_1.png     PNG File NCBC-649_2.png     PNG File NCBC-649_3.png     PNG File NCBC-649_4.png     File NCBC-649.7z     File NCBC-649_gerrit.7z     PNG File NCBC-649.png    

 Description   
run with commit https://github.com/couchbase/couchbase-net-client/commit/24714876f76207a5fc7d72e88bd1f9c57909fcdb

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/102/console

Running ReAdd2-HYBRID: 2014-09-18 23:50:57
    workload=HybridWorkloadGroup
    failover/next_action=FO_READD_REBALANCE
    failover/count=2
    failover/next_delay=45
    testcase=FailoverScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\ReAdd2-HYBRID\09-18-14\024657\8c847f52905fd0bf669803f16c2dc716

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
        --workload HybridWorkloadGroup --failover-next_action
 \
        FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
 \
        --testcase FailoverScenario -C share\rexec --rexec_path
 \
        C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \
        --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \
        --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \
        --cluster_ssh-username=root --cluster_ssh-password=couchbase


2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,614 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 1 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,614 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#878 on thread 42
2014-09-18 23:56:46,614 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 609 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 - rev878
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP4463REP4463REP4463REP4463REP4463 using server 10.3.121.134:11210
2014-09-18 23:56:46,615 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b | 10.3.121.134:11210 | [0, 2] - b20c68a7-f922-486e-af16-f880d88bfff6 - Disposed: False
2014-09-18 23:56:46,615 [42] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP4463REP4463REP4463REP4463REP4463 using 609 on 10.3.121.134:11210
2014-09-18 23:56:46,616 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b on 10.3.121.134:11210 - b20c68a7-f922-486e-af16-f880d88bfff6
2014-09-18 23:56:46,619 [42] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1736
2014-09-18 23:56:46,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:46,880 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 2 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 failed after 3 retries. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:49,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:51,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:54,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:56,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:59,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:01,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:04,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


..................


2014-09-19 01:09:51,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:54,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:56,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:59,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:10:01,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


test stuck, not all items loaded
run was via jenkins, so I'm not able to get stacktraces



 Comments   
Comment by Jeff Morris [ 19/Sep/14 ]
Hmm, looks like connection pool exhausted, but none of the in-progress operations are timing out? Strange.
Comment by Andrei Baranouski [ 19/Sep/14 ]
right, something like that
Comment by Andrei Baranouski [ 20/Sep/14 ]
catched it in manual run


  [In a sleep, wait, or join]
  mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) + 0x22 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) + 0x28 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout) + 0xd bytes
> Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 123 + 0x28 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
 
        .................................
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Couchbase.IO.IConnectionPool.Acquire() Line 178 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.Strategies.DefaultIOStrategy.Execute<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 40 + 0xc bytes C#
  Couchbase.dll!Couchbase.Core.Server.Send<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 88 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 103 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.Upsert<object>(string key, object value) Line 332 + 0x31 bytes C#
  Sdkd.dll!Sdkd.Commands.SetCommand.ExecuteSingle(string key, string value) Line 78 + 0x1e bytes C#
  Sdkd.dll!Sdkd.Commands.KVCommand.Run() Line 24 + 0x36 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.Buckets.CouchbaseBucket cli) Line 266 + 0x9 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 146 + 0xf bytes C#
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x6f bytes
  mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0xa7 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x16 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x41 bytes
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes
  [Native to Managed Transition]


please note, this run was with some commits from gerrit

$ git log
commit 8afa0ace0ac366e53dcc07a62c23ffd6604f6015
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

    Change-Id: I41bd3d93b31b98a19b8fb23d432713f5034debd7

commit 602181eed21614451ccaf0f848337f9c382ae578
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 00:10:53 2014 -0700

    NCBC-590: Refactor logging so that ODE are not raised when it references a disposed object

    Change-Id: I5d81dba00d64e4caf34d165491624e2c02a9949b

commit f9d4efb511c93a64b2a844e41849177da1f4442a
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:53:12 2014 -0700

    Refactor unit tests to reflect changes in exceptions when bootstrap fails

    Change-Id: I387aa3c8a0d8954102265e27dbbe4f6d84c4d104

commit 2a054aceab491ed338f1fc435d0a48eb4d31c43c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:13:02 2014 -0700

    NCBC-624: Refactor View API

    From Michael's review:

    "One thing that I was confused about was the View query DSL. The required
    argument is development, but the view and design document name could be
    omitted by the DSL user since they are elements like limit and so forth. I
    think it would be better to turn this around. For example in Java there is
    ViewQuery.from('designDoc', 'view') and then the DSL starts.
    So you can never miss design doc and view name by accident. And I think
    you can go with a development = false default, since that's what people
    use most of the time. I think the main question is, if I miss adding
    design doc or view name by accident, does it blow up at runtime or at
    compile time?"

    Change-Id: I4fc9d716f674df882156cdd1763b8d43204e7d90

commit 24714876f76207a5fc7d72e88bd1f9c57909fcdb
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 19:06:31 2014 -0700

    NCBC-606: Catch and log ServerUnavailableException during failover

    Change-Id: Ie0f74d2bf69cd43ea399762fc39e7ca7f305c11c
    Reviewed-on: http://review.couchbase.org/41384
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit a2cd618f125fc330db742bcb6879c85741e1b3e0
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:43:48 2014 -0700

    NCBC-638: If Server is null when sending an operation then retry operation

    Change-Id: Idfee0ab2852c51cb9c7de1912374d64ee1633be7
    Reviewed-on: http://review.couchbase.org/41374
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 2ca77982d595f15cfa7fe5aa0b97902c94b5f120
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:41:13 2014 -0700

    Update unit tests to use camel cased property values when applicable

    Change-Id: I80ae31e2bf58809574dba965063b21ffe48dc5ec
    Reviewed-on: http://review.couchbase.org/41373
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 33fc515c0248fbeb56fa7123cea954872e4c9b11
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:35:10 2014 -0700

    NCBC-637: When CCCP fails ensure NRE is not thrown

    Change-Id: I5ef0dbef8298675ceb92e9eb91c49303269133a2
    Reviewed-on: http://review.couchbase.org/41371
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 1e9fff367fa3eefd19069fdfa1dfaae14b4b2a42
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 23:36:43 2014 -0700

    NCBC-586: Make AddRef and Release on IBucket not public

    This commit adds an internal interface IRefCountable with AddRef and
    Release members, so that the two methods are not publically callable on an
    IBucket implementation.

    Change-Id: I11f628094aa1f65ff4780f42e00bbfb088e78898
    Reviewed-on: http://review.couchbase.org/41354
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 983b94b8acbce8eee11314542d8dcdfbb3c03e0d
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 22:34:37 2014 -0700

    NCBC-636: Make outgoing ContractResolver default to camel casing

    This commit makes the incoming and outgoing JSON IContractResolver's type
    configurable from the ClientConfiguration object. The default settings are
    to use a CamelCasedNameContractResolver for both serialization and
    deserilization.

    Change-Id: I3a82f95d5f4608aa8d05e4141a1a089bc300d4e7
    Reviewed-on: http://review.couchbase.org/41353
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 77b78aaa1550c2ea9e0dfa4085fad2cd24814721
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 18:06:55 2014 -0700

    NCBC-635: Provide better failure support for N1QL Query API

    Change-Id: I16d9a44ba653f8846b22e7c679681ab262e1e6bc
    Reviewed-on: http://review.couchbase.org/41351
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 424a10647f5d6dd21e603fbfe8bb98de305f9491
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:43:24 2014 -0700

    NCBC-634: Ensure that CancellationTokenSource is Disposed

    This fix ensures that the CTS will be explicitly disposed so that the
    object does not live through to finalization.

    Change-Id: I39545b97a3c934f2c0a93587fa7147b32d2bd043
    Reviewed-on: http://review.couchbase.org/41335
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8bd607e8cb545ace5102edf1accb5f862b5caf74
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:12:17 2014 -0700

    NCBC-632: Make JsonSerializerSettings configurable

    Change-Id: I21c5620a2192217faa68a62cf440a2a8f3dc8893
    Reviewed-on: http://review.couchbase.org/41331
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit bebae67219460e2e3a69ad2d56c9f98afc824190
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 14:26:25 2014 -0700

    NCBC-626: HttpRequestException in queries after rebalance out

    Change-Id: I0eed00d956f231c3826a54da6a983586618d126d
    Reviewed-on: http://review.couchbase.org/41327
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8db6085e9aecb23cd6defb672f7ec0548536d9de
Author: jeffrymorris <jeffrymorris@gmail.com>

http://www.couchbase.com/issues/secure/attachment/21870/NCBC-649_gerrit.7z


please see screenshots
Comment by Andrei Baranouski [ 26/Sep/14 ]
still see the issue with https://github.com/couchbase/couchbase-net-client/commit/72b06be3aa7b6ae8ed478f006fd7ebe89f3cae89
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/106/console 2.5.0-1059
Comment by Andrei Baranouski [ 30/Sep/14 ]
unable to reproduce with https://github.com/couchbase/couchbase-net-client/commit/747fa5523cf0b8cbefb9f615b877c41136b4fec7




.NET SDK 2.0 Documentation (NCBC-516)

[NCBC-495] Code Comments & Sandcastle Documentation Created: 02/Jun/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Closed
Project: Couchbase .NET client library
Component/s: docs
Affects Version/s: 2.0 DP1
Fix Version/s: 2.0 Beta

Type: Technical task Priority: Blocker
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Jeff Morris [ 17/Jun/14 ]
http://www.couchbase.com/issues/browse/NCBC-519




[NCBC-265] Refactor solution so that Enyim is part of main source branch Created: 20/May/13  Updated: 02/Oct/13  Resolved: 02/Oct/13

Status: Closed
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 1.2.6
Fix Version/s: 1.2.7

Type: Improvement Priority: Blocker
Reporter: John Zablocki (Inactive) Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Blocker, because it's required before next release.




[NCBC-161] Views failing when not all nodes contain couchApiBase Created: 22/Nov/12  Updated: 07/Dec/12  Resolved: 07/Dec/12

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 1.2 Beta-2
Fix Version/s: 1.2.0

Type: Bug Priority: Blocker
Reporter: Mark Nunberg Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Dies with IllegalOperationException (or some such)..

 Comments   
Comment by Mark Nunberg [ 22/Nov/12 ]
http://review.couchbase.org/22770
Comment by John Zablocki (Inactive) [ 07/Dec/12 ]
released




[NCBC-158] Raise view timeout from 60s to 75s Created: 21/Nov/12  Updated: 21/Nov/12  Resolved: 21/Nov/12

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 1.2 Beta
Fix Version/s: 1.2 Beta-2

Type: Improvement Priority: Blocker
Reporter: Matt Ingenthron Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Since the server side view timeout is 60s, we should raise the client side a bit higher. 75s seems like the right number. It should be tuneable though.

 Comments   
Comment by John Zablocki (Inactive) [ 21/Nov/12 ]
Published




[NCBC-137] design document management is not in the documentation or API reference Created: 05/Oct/12  Updated: 08/Jan/13  Resolved: 08/Jan/13

Status: Closed
Project: Couchbase .NET client library
Component/s: docs
Affects Version/s: None
Fix Version/s: 1.2.1

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Had reason to look up design document management from .NET earlier today and it seems to not be documented. This and all new features should be added to the API reference before release.

 Comments   
Comment by John Zablocki (Inactive) [ 08/Jan/13 ]
http://www.couchbase.com/docs/couchbase-sdk-net-1.2/couchbase-sdk-net-cluster-management.html
Comment by John Zablocki (Inactive) [ 08/Jan/13 ]
Merged into docs.




[NCBC-134] Failover on bootstrap node never recovers Created: 02/Oct/12  Updated: 03/Jan/13  Resolved: 21/Nov/12

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 1.1.6
Fix Version/s: 1.2 Beta-2

Type: Bug Priority: Blocker
Reporter: Mark Nunberg Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Server 2.0.

Attachments: Text File dotnet.log     Zip Archive dotnet-log.zip    

 Description   
When I fail over two nodes while performing operations, where one of them it the 'entry point' node (i.e. the one which is the source of the streaming API), it seems the client never recovers. It does not crash or provide an exception. Operations continue failing from the workload generator.

This is using a modified Enyim (see NCBC-121) without which I'm not even trying as it seems to crash my application during the slightest network issue.

 Comments   
Comment by Mark Nunberg [ 25/Oct/12 ]
I've attached the log file - it's rather large :)
Comment by John Zablocki (Inactive) [ 21/Nov/12 ]
Published




[NCBC-28] support new stale=false parameter Created: 01/Mar/12  Updated: 07/Dec/12  Resolved: 02/Mar/12

Status: Closed
Project: Couchbase .NET client library
Component/s: docs, library
Affects Version/s: None
Fix Version/s: 1.2 DP

Type: Improvement Priority: Blocker
Reporter: Matt Ingenthron Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Starting with 2.0 DP4, the default for stale has changed to update_after and there is a new argument to the stale parameter, "false" which performs a view update before response. The client needs to support this new argument to the stale parameter.

 Comments   
Comment by John Zablocki (Inactive) [ 02/Mar/12 ]
Support for stale=false submitted for review in Gerrit




[NCBC-121] Null Reference Exception during rebalance Created: 17/Sep/12  Updated: 03/Jan/13  Resolved: 21/Nov/12

Status: Closed
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 1.1.6
Fix Version/s: 1.2 Beta-2

Type: Bug Priority: Blocker
Reporter: Mark Nunberg Assignee: John Zablocki (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File net-bug.png    

 Description   
Happens during the rebalancing-out of two nodes.

Reproduce:

* Perform get/set operations continuously
* Rebalance out two nodes
poof!

I was using multiple client handles for this test.. it seems each of them got that error.

 Comments   
Comment by Farshid Ghods (Inactive) [ 17/Sep/12 ]
mark,

Just wondering if major is the right priority for this issue given that this happens during rebalancing
Do users tend to ignore one or two errors and are okay with retries for these type of errors
Comment by Mark Nunberg [ 17/Sep/12 ]
This causes a complete application crash.
Comment by Mark Nunberg [ 19/Sep/12 ]
I've patched up Enyim a bit so that it no longer crashes -- the problem being some apparent reentrancy and race condition issues when a node is dying.

I'll file a gerrit patch when I figure out how to use this thing on my windows box
Comment by Mark Nunberg [ 27/Sep/12 ]
Interim changeset is here
https://github.com/mnunberg/EnyimMemcached/commit/f2b161a0909badf7981e055605d3a20f9a5db4dd
Comment by Matt Ingenthron [ 04/Oct/12 ]
John, can you evaluate the severity of this? This has been a blocker for QE tests.
Comment by John Zablocki (Inactive) [ 23/Oct/12 ]
Merged Mark's code into our Enyim fork
Comment by John Zablocki (Inactive) [ 21/Nov/12 ]
Published




[NCBC-554] The remote certificate is invalid according to the validation procedure. Created: 18/Jul/14  Updated: 23/Jul/14  Resolved: 23/Jul/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 DP3

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: PNG File cert_screen.png     Text File log.txt     Text File test_logs.txt    
Sub-Tasks:
Key
Summary
Type
Status
Assignee
NCBC-556 SslConnection uses obsolete GetBuffer... Technical task Closed Jeff Morris  

 Description   
as a continuation of the NCBC-553 ticket

in my tests i set use ssl for ClientConfiguration & BucketConfiguration

var bucketConfiguration = new BucketConfiguration();
bucketConfiguration.UseSsl = true;
var config = new ClientConfiguration();
config.UseSsl=true;

Then I've install cluster's certificate on vm where I run my tests(see sceenshot)

when I run my tests I get

client logs:

2014-07-16 16:19:16,768 [4] DEBUG Couchbase.Core.ClusterManager - !!Trying to boostrap with Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.
2014-07-16 16:19:16,950 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-07-16 16:19:17,041 [4] WARN Couchbase.Core.ClusterManager - System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost)
   at Couchbase.IO.Strategies.SslConnection.Authenticate() in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\SslConnection.cs:line 34
   at Couchbase.IO.DefaultConnectionFactory.<GetGeneric>b__4[T](IConnectionPool`1 p, IByteConverter c) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\DefaultConnectionFactory.cs:line 69
   at Couchbase.IO.ConnectionPool`1.Acquire() in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\ConnectionPool.cs:line 108
   at Couchbase.IO.ConnectionPool`1.Couchbase.IO.IConnectionPool.Acquire() in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\ConnectionPool.cs:line 170
   at Couchbase.Authentication.SASL.SaslFactory.<GetFactory3>b__0(String username, String password, IOStrategy strategy, IByteConverter converter) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Authentication\SASL\SaslFactory.cs:line 21
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 31
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:line 149


test logs:

 c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0.34 INFO] (BRun run:435) Initializing history database
Will create SASL????true
Will create SASL????true
Will create SASL????true
Will create SASL????true

============================================================
Running Rb2Out-HYBRID
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-933\Rb2Out-HYBRID\07-16-14\083910\444e6acd28e9170d8ce86bfc626d8674

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase
        --cluster_useSSL=True
    ---------------------------------------->8
[1.58 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
Will create SASL????true
Will create SASL????true
Will create SASL????true
Will create SASL????true
[2.05 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[2.07 INFO] (RunContext run:124) Starting cluster and driver
[2.08 INFO] (CBCluster startCluster:360) Node http://10.3.121.134:8091 is master now
[2.08 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[2.08 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2.44 INFO] (SDKD log:137) --- Logging Self-Test ---
[2.46 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[2.46 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[2.47 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[2.47 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[2.47 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2.62 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2.63 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2.92 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.17929"}}
[3.33 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.135:8091,ns_1@10.3.121.135>
[3.36 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.136:8091,ns_1@10.3.121.136>
[3.39 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.3.206:8091,ns_1@10.3.3.206>
[3.49 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[3.50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[3.50 DEBUG] (SSHConnection initUserKeys:79) Loaded indentity C:\Users\Administrator\.ssh\id_rsa
[3.51 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.06 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.134:8091
[4.07 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.135:8091
[4.07 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.136:8091
[4.07 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.3.206:8091
[4.09 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.136
[4.09 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.134
[4.10 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.135
[4.10 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.3.206
[4.30 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@174323d5
[4.31 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@f5d8d75
[4.31 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@3af2164f
[4.32 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@74bfd10a
[4.32 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@5d684e26
[25.89 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.135:8091
[31.16 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.136:8091
[34.06 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.3.206:8091
[39.84 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[39.87 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[40.88 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[40.88 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[40.89 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[40.90 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[47.18 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[47.22 INFO] (RunContext run:143) Driver and cluster initialized
[47.24 INFO] (RunContext call:167) Running scenario..
[47.24 INFO] (Scenario run:72) Starting RAMP phase
[47.24 INFO] (Workload setupDesign:63) Creating design test_design
[47.50 INFO] (Workload setupDesign:80) Design creation done
[47.50 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=true, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["
0.3.3.206",8091]]}, Hostname=10.3.121.134}
[47.51 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[47.51 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[47.52 INFO] (SDKD log:137) ClientConfiguration!!!!!!!!!!!!{
[47.52 INFO] (SDKD log:137) "Port": 8091,
[47.52 INFO] (SDKD log:137) "Bucket": "default",
[47.53 INFO] (SDKD log:137) "Options": {
[47.53 INFO] (SDKD log:137) "SSL": true,
[47.53 INFO] (SDKD log:137) "Username": "default",
[47.53 INFO] (SDKD log:137) "Password": "",
[47.54 INFO] (SDKD log:137) "OtherNodes": [
[47.54 INFO] (SDKD log:137) [
[47.54 INFO] (SDKD log:137) "10.3.121.135",
[47.55 INFO] (SDKD log:137) 8091
[47.55 INFO] (SDKD log:137) ],
[47.55 INFO] (SDKD log:137) [
[47.55 INFO] (SDKD log:137) "10.3.121.136",
[47.56 INFO] (SDKD log:137) 8091
[47.56 INFO] (SDKD log:137) ],
[47.56 INFO] (SDKD log:137) [
[47.56 INFO] (SDKD log:137) "10.3.3.206",
[47.57 INFO] (SDKD log:137) 8091
[47.57 INFO] (SDKD log:137) ]
[47.57 INFO] (SDKD log:137) ]
[47.57 INFO] (SDKD log:137) },
[47.58 INFO] (SDKD log:137) "Hostname": "10.3.121.134"
[47.58 INFO] (SDKD log:137) }
[47.58 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[48.13 INFO] (SDKD log:137)
[48.14 INFO] (SDKD log:137) Unhandled Exception: Couchbase.Configuration.ConfigException: Could not bootstrap default. See log for details.
[52.62 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[52.62 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.2
[52.62 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterMana
er.cs:line 204
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName) in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:line 137
   at Couchbase.CouchbaseCluster.OpenBucket() in c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\CouchbaseCluster.cs:line 70
   at Sdkd.Couchbase.ClientFactory.FindExistingClient(ClientConfiguration config) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 77
   at Sdkd.Couchbase.ClientFactory.CreateClient(ClientConfiguration config) in c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 256
   at SdkdConsole.Daemon.Handle.CreateHandle(Request req) in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 84
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 103
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[52.64 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]
[52.80 ERROR] (BRun main:507) Caught exception
com.couchbase.sdkdclient.context.HarnessException: java.net.SocketException: Connection reset
        at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:80) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:169) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:165) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
        at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[na:1.6.0_27]
        at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[na:1.6.0_27]
        at java.io.InputStreamReader.read(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.fill(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.6.0_27]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.6.0_27]
        at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:152) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.sendMessage(Handle.java:173) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.negotiate(Handle.java:237) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.HandleFactory.createHandle(HandleFactory.java:43) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.ViewWorkload.loadData(ViewWorkload.java:85) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.HybridWorkloadGroup.start(HybridWorkloadGroup.java:82) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:78) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        ... 10 common frames omitted


Jeff, I think I have correctly installed the certificate, but why do I get this error


 Comments   
Comment by Andrei Baranouski [ 21/Jul/14 ]
another attempt:

before creating the connection programmatically install a certificate:
            string file = "C:\\temp\\certificate"; // Contains name of certificate file
            X509Store store = new X509Store(StoreName.Root, StoreLocation.CurrentUser);
            store.Open(OpenFlags.ReadWrite);
            store.Add(new X509Certificate2(X509Certificate2.CreateFromCertFile(file)));
            store.Close();




client logs:

2014-07-21 12:19:53,172 [4] DEBUG Couchbase.Core.ClusterManager - !!Trying to boostrap with Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.
2014-07-21 12:19:53,432 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-07-21 12:19:53,981 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: f513fedf-af29-43c7-aafe-bd7811a46038 - [0, 0]
2014-07-21 12:19:53,996 [4] ERROR Couchbase.Authentication.SASL.SaslFactory - System.NotImplementedException: The method or operation is not implemented
   в Couchbase.IO.Operations.OperationBase`1.GetBuffer() в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Operations\OperationBase'.cs:line 168
   в Couchbase.IO.Strategies.SslConnection.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\SslConnection.cs:line 40
   в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation, IConnection connection) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 32
   в Couchbase.Authentication.SASL.SaslFactory.<GetFactory3>b__0(String username, String password, IOStrategy strategy, IByteConverter converter) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Authentication\SASL\SaslFactory.cs:line 24
2014-07-21 12:19:54,243 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-07-21 12:19:54,737 [4] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: 3cc96df3-2534-4c79-a5c3-12e2a635b156 - [0, 1]


test logs:

[46,24 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[46,49 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[46,75 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[52,35 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[55,24 INFO] (RunContext run:143) Driver and cluster initialized
[55,51 INFO] (RunContext call:167) Running scenario..
[55,51 INFO] (Scenario run:72) Starting RAMP phase
[55,51 INFO] (Workload setupDesign:63) Creating design test_design
[56,81 INFO] (Workload setupDesign:80) Design creation done
[56,81 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[56,81 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=true, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[56,82 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[62,01 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[62,01 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[62,01 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[62,01 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[62,02 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[63,69 INFO] (SDKD log:137)
[63,70 INFO] (SDKD log:137) Unhandled exception: System.NotImplementedException: The method or operation is not implemented
[63,71 INFO] (SDKD log:137) в Couchbase.IO.Operations.OperationBase`1.GetBuffer() в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Operations\OperationBase'.cs:line 168
[63,71 INFO] (SDKD log:137) в Couchbase.IO.Strategies.SslConnection.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\SslConnection.cs:line 40
[63,71 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation, IConnection connection) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 32
[63,71 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 42
[63,72 INFO] (SDKD log:137) в Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Configuration\S
erver\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 35
[63,72 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:line 149
[63,72 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.CreateBucket(String bucketName) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:line 137
[63,73 INFO] (SDKD log:137) в Couchbase.CouchbaseCluster.OpenBucket() в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\CouchbaseCluster.cs:line 70
[63,73 INFO] (SDKD log:137) в Sdkd.Couchbase.ClientFactory.FindExistingClient(ClientConfiguration config) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 78
[63,73 INFO] (SDKD log:137) в Sdkd.Couchbase.ClientFactory.CreateClient(ClientConfiguration config) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 264
[63,74 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.CreateHandle(Request req) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 84
[63,74 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 103
[63,74 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[63,75 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[63,75 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[63,75 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[63,76 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart()
[71,69 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.2
[71,71 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]
[71,97 ERROR] (BRun main:507) Caught exception
com.couchbase.sdkdclient.context.HarnessException: java.net.SocketException: Connection reset
        at com.couchbase.sdkdclient.context.HarnessException.create(HarnessException.java:56) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:80) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:169) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$4.call(RunContext.java:165) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60]
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.7.0_60]
        at java.net.SocketInputStream.read(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamDecoder.read(Unknown Source) ~[na:1.7.0_60]
        at java.io.InputStreamReader.read(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedReader.fill(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedReader.readLine(Unknown Source) ~[na:1.7.0_60]
        at com.couchbase.sdkdclient.handle.Handle.receiveMessage(Handle.java:152) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.sendMessage(Handle.java:173) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.negotiate(Handle.java:237) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.HandleFactory.createHandle(HandleFactory.java:43) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.ViewWorkload.loadData(ViewWorkload.java:85) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.workload.HybridWorkloadGroup.start(HybridWorkloadGroup.java:82) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.scenario.PhasedScenario.run(PhasedScenario.java:78) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        ... 8 common frames omitted

Comment by Jeff Morris [ 21/Jul/14 ]
https://www.couchbase.com/issues/browse/NCBC-556
Comment by Andrei Baranouski [ 23/Jul/14 ]
Hi Jeff,

I need to reopen the issue because I still can't use system installed certificate

I use the latest version https://github.com/couchbase/couchbase-net-client/commit/aa078fef70d4be711b704e522611f71d229c98c9

2014-07-23 19:15:06,985 [4] DEBUG Couchbase.Core.ClusterManager - Trying to boostrap with Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.
2014-07-23 19:15:07,239 [4] WARN Couchbase.IO.ConnectionBase - Starting SSL encryption on 10.3.121.134
2014-07-23 19:15:08,447 [4] ERROR Couchbase.IO.ConnectionBase - System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
  inSystem.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
  inSystem.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)

all logs attached

https://www.couchbase.com/issues/secure/attachment/21366/log.txt
https://www.couchbase.com/issues/secure/attachment/21367/test_logs.txt
Comment by Jeff Morris [ 23/Jul/14 ]
Andrei -

Is the certificate installed correctly and up-to-date?

-Jeff
Comment by Andrei Baranouski [ 23/Jul/14 ]
Seems like installation cert via IE doesn't work properly(Brett's steps)
manual steps from Jeff that works for me:


Here are the steps I use:
[19:31:07] Jeff Morris: 1) Copy the cert from the settings tab and paste it into notepad
[19:32:01] Jeff Morris: 2) Save it to C: (root) as [somecertname].cert
[19:33:05] Jeff Morris: 3) Open the windows cert mgr:
      call certmgr.msc in cmd
[19:34:01] Jeff Morris: 4) right click on the trusted root certification authority and then "import"
[19:34:17] Jeff Morris: all tasks->import
[19:35:01] Jeff Morris: 5) click next and then use the dialog to find the cert stored at "C:"
[19:35:15] Jeff Morris: then close the dialogue

so, I get the same certificate if I install it programmatically
string file = "C:\\temp\\certificate"; // Contains name of certificate file
            X509Store store = new X509Store(StoreName.Root, StoreLocation.CurrentUser);
            store.Open(OpenFlags.ReadWrite);
            store.Add(new X509Certificate2(X509Certificate2.CreateFromCertFile(file)));
            store.Close();

 




[NCBC-536] [.net client 2.0]ConfigObserverNotFoundException during rebalance + mutation Created: 26/Jun/14  Updated: 21/Aug/14  Resolved: 09/Jul/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 DP3

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File App.config    
Issue Links:
Dependency

 Description   
I run the test agains 2.5 and used not yet commited sdkd-net(client-2.0) version

packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args -d all:debug --no-upload --dsw_kvcount 10


[47,87 INFO] (SDKD log:137) SetCommand req.Command MC_DS_MUTATE_SET
[47,87 INFO] (SDKD log:137) SetCommand req.Command MC_DS_MUTATE_SET
[77,76 INFO] (Scenario run:86) Starting CHANGE phase
[78,22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[79,88 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[81,11 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 2%
[82,35 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 4%
[83,58 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 5%
[84,80 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 7%
[86,03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 8%
[87,26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10%
[88,49 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 11%
[88,66 INFO] (SDKD log:137)
[88,68 INFO] (SDKD log:137) Необработанное исключение: Couchbase.Configuration.ConfigObserverNotFoundException: default
[88,68 INFO] (SDKD log:137) в Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.UpdateConfig(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Configuration\Server
\Providers\CarrierPublication\CarrierPublicationProvider.cs:строка 78
[88,69 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.NotifyConfigPublished(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:строка 196
[88,69 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 464
[88,69 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 86
[88,69 INFO] (SDKD log:137) в Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:строка 77
[88,70 INFO] (SDKD log:137) в Sdkd.Commands.KVCommand.Run() в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:строка 25
[88,70 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.DispatchCommand(Request req) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 193
[88,70 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 125
[88,70 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[88,70 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[88,71 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[88,71 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[88,71 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart()
[89,71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 12%
[90,94 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 14%
[92,17 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 15%
[93,40 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17%
[94,71 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 18%
[94,74 ERROR] (BRun maybePropagate:236) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60]
[94,75 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.41
[94,75 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:253) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:275) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:382) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:466) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:508) [sdkdclient-1.0-SNAPSHOT.jar:na]


Sorry, some Russian words :)



[21:37:47] Brett Lawson: Yea, it LOOKS like it could be a .NET bug, but it also could be a bug in the test.
[21:38:32] Brett Lawson: You should debug a little bit to try and identify how its occuring.
[21:39:52] Andrei Baranouski: okay, will post a ticket for now. then I will commit my sdkd fixes and will provide steps to reproduce (command line)


 Comments   
Comment by Andrei Baranouski [ 27/Jun/14 ]
some investigations:

I've the following changes in GetCommand.cs & SetCommand.cs to handle ConfigObserverNotFoundException when rebalance started
If i get ConfigObserverNotFoundException I sleep 150 sec to wait when rebalance completed and retry:



$ git diff
diff --git a/src/Sdkd/Commands/GetCommand.cs b/src/Sdkd/Commands/GetCommand.cs
index 55a90eb..588f1d0 100644
--- a/src/Sdkd/Commands/GetCommand.cs
+++ b/src/Sdkd/Commands/GetCommand.cs
@@ -8,6 +8,8 @@ using Sdkd.Protocol.Dataset;
 using Couchbase.Core;
 using Couchbase.IO.Operations;
 using Couchbase.Core.Buckets;
+using Couchbase.Configuration;^M
+using System.Threading;^M

 namespace Sdkd.Commands
 {
@@ -15,7 +17,22 @@ public class GetCommand : KVCommand
 {
        public override IOperationResult<Object> ExecuteSingle(string key, string value)
        {
- IOperationResult<String> res = Client.Get<String>(key);
+ Console.WriteLine(" AAAA " + Client.Name);^M
+ Console.WriteLine(Client.ToString());^M
+ IOperationResult<String> res = null;^M
+ try^M
+ {^M
+ res = Client.Get<String>(key);^M
+ }^M
+ catch (ConfigObserverNotFoundException e)^M
+ {^M
+ Console.WriteLine("UNABLE TO GET " + key);^M
+ Console.WriteLine(e.StackTrace);^M
+ Thread.Sleep(150000);^M
+ res = Client.Get<String>(key);^M
+ Console.WriteLine("SUCCESS TO GET " + key);^M
+ }^M
+ ^M
                if (res.Success == false)
                {
                        // Do something here


diff --git a/src/Sdkd/Commands/SetCommand.cs b/src/Sdkd/Commands/SetCommand.cs
index 63d7091..466741d 100644
--- a/src/Sdkd/Commands/SetCommand.cs
+++ b/src/Sdkd/Commands/SetCommand.cs
@@ -9,6 +9,8 @@ using Newtonsoft.Json.Linq;
 using Couchbase.Core;
 using Couchbase.IO.Operations;
 using Couchbase.Core.Buckets;
+using Couchbase.Configuration;^M
+using System.Threading;^M

 namespace Sdkd.Commands
 {
@@ -74,7 +76,21 @@ public class SetCommand : KVCommand
         }
         else if (Mode == OperationCode.Set)
         {
- return Client.Upsert<Object>(key, value);
+ try^M
+ {^M
+ return Client.Upsert<Object>(key, value);^M
+ }^M
+ catch (ConfigObserverNotFoundException e)^M
+ {^M
+ Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt"));^M
+ Console.WriteLine("UNABLE TO UPSERT " + key);^M
+ Console.WriteLine(e.StackTrace);^M
+ Thread.Sleep(150000);^M
+ Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt"));^M
+ IOperationResult<Object> temp = Client.Upsert<Object>(key, value);^M
+ Console.WriteLine("SUCCESS TO UPSERT " + key);^M
+ return temp;^M
+ }^M
         }
         else
         {


[89,82 INFO] (SDKD log:137) UNABLE TO UPSERT SimpleKeyREP3REP3REP3REP3REP3REP3REP3REP3
[89,82 INFO] (SDKD log:137) в Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.UpdateConfig(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Configuration\Server
\Providers\CarrierPublication\CarrierPublicationProvider.cs:строка 78
[89,82 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.NotifyConfigPublished(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:строка 196
[89,82 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 464
[89,82 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 86
[89,83 INFO] (SDKD log:137) в Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:строка 81
[90,74 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 22%
[91,97 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 23%
[93,19 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 24%
......................
[199,12 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[200,34 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[201,56 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[201,57 INFO] (Scenario run:97) CHANGE phase done
[201,57 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[230,00 INFO] (SDKD log:137) 11:28:49
[230,01 INFO] (SDKD log:137) SUCCESS TO UPSERT SimpleKeyREP3REP3REP3REP3REP3REP3REP3REP3
[230,67 INFO] (SDKD log:137) 11:28:49
[230,67 INFO] (SDKD log:137) UNABLE TO UPSERT SimpleKeyREP5REP5REP5REP5REP5REP5REP5REP5
[230,67 INFO] (SDKD log:137) в Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.UpdateConfig(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Configuration\Serve
r\Providers\CarrierPublication\CarrierPublicationProvider.cs:строка 78
[230,67 INFO] (SDKD log:137) в Couchbase.Core.ClusterManager.NotifyConfigPublished(IBucketConfig bucketConfig) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\ClusterManager.cs:строка 196
[230,67 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 464
[230,68 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 86
[230,68 INFO] (SDKD log:137) в Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:строка 81
[231,09 INFO] (SDKD log:137) 11:28:50
[231,10 INFO] (SDKD log:137)
[231,11 INFO] (SDKD log:137) Необработанное исключение: System.IO.IOException: Не удается записать данные в транспортное соединение: Программа на вашем хост-компьютере разорвала установленное подключение. ---> System.Net.Sockets.SocketException: Прог
рамма на вашем хост-компьютере разорвала установленное подключение
[231,11 INFO] (SDKD log:137) в System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
[231,11 INFO] (SDKD log:137) в System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
[231,11 INFO] (SDKD log:137) --- Конец трассировки внутреннего стека исключений ---
[231,12 INFO] (SDKD log:137) в System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
[231,12 INFO] (SDKD log:137) в Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\EapConnection.cs:строка 35
[231,12 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation, IConnection connection) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 32
[231,12 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 42
[231,13 INFO] (SDKD log:137) в Couchbase.Core.Server.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Server.cs:строка 80
[231,13 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 84
[231,13 INFO] (SDKD log:137) в Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:строка 90
[231,13 INFO] (SDKD log:137) в Sdkd.Commands.KVCommand.Run() в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:строка 25
[231,13 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.DispatchCommand(Request req) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 191
[231,14 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 123
[231,14 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[231,14 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[231,14 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[231,15 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[231,15 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart()


so, here I see that after rebalance the second attempt was successful for SimpleKeyREP3REP3REP3REP3REP3REP3REP3REP3 but I again get exception for SimpleKeyREP5REP5REP5REP5REP5REP5REP5REP5
Comment by Andrei Baranouski [ 30/Jun/14 ]
my final conclusion after a deep study of the problem

when we perform Upset item during rebalance we call Upsert() and then check operationResult.Status == ResponseStatus.VBucketBelongsToAnotherServer
so, in the tests we get _VBucketBelongsToAnotherServer



CouchbaseBucket.sh:
 public IOperationResult<T> Upsert<T>(string key, T value)
        {
            IVBucket vBucket;
            var server = GetServer(key, out vBucket);

            var operation = new SetOperation<T>(key, value, vBucket, _converter);
            var operationResult = server.Send(operation);

            if (CheckForConfigUpdates(operationResult)) --- CHECK RESULT
            {
                Log.Info(m => m("Requires retry {0}", key));
            }
            return operationResult;
        }



 bool CheckForConfigUpdates<T>(IOperationResult<T> operationResult)
        {
            var requiresRetry = false;
            if (operationResult.Status == ResponseStatus.VBucketBelongsToAnotherServer)
            {
                var bucketConfig = ((OperationResult<T>)operationResult).GetConfig();
                if (bucketConfig != null)
                {
                    Log.Info(m => m("New config found {0}", bucketConfig.Rev));
                    _clusterManager.NotifyConfigPublished(bucketConfig); ------------------------FAILED because ConfigObservers is empty for this bucketConnection
                    requiresRetry = true;
                }
            }
            return requiresRetry;
        }


I tried to find the problem in the tests, as code may not yet be _clean after switching on 2.0, but apparently the problem on the SDK client side, because I have no problems for CRUD operation with CouchbaseBucket instance, but when rebalance starts....
Comment by Andrei Baranouski [ 30/Jun/14 ]
Jeff, could you take a look at the issue?
if you need any assistance (details steps to reproduce or create separate ticket in 'Couchbase .NET client library", ...) please let me know.
Comment by Andrei Baranouski [ 30/Jun/14 ]
maybe my test logs will be something useful https://friendpaste.com/2mIKMKUmN9rTRdUUX5fKTf
Comment by Jeff Morris [ 30/Jun/14 ]
Andrei -

Thanks for the detailed analysis, I'll take a look into it.

- Jeff
Comment by Jeff Morris [ 08/Jul/14 ]
Andrei -

Is this happening on every run or is this the initial test run? Having a bit of a hard time replicating it, although I don't doubt it's existence :)

-Jeff
Comment by Andrei Baranouski [ 09/Jul/14 ]
yes, it happens in every test. but not immediately, it happens when the tests trigger rebalance and vbucket map is changed
Comment by Jeff Morris [ 09/Jul/14 ]
Change the value of this element to a valid path on the machine running SDK: <param name="File" value="C:\temp\log.txt" />

This file also has to be included in the SDKD directory - or cut and paste the relevant bits so that logging is enabled.
Comment by Jeff Morris [ 09/Jul/14 ]
commit 369b8f312239156f9a46fba279afd7c5a84b3c77

Use the contents of the app.config attached to enable logging and run the tests again. Afterwards, attach the log file so I can review it.
Comment by Andrei Baranouski [ 10/Jul/14 ]
Thanks Jeff! seems like the issue has been resolved http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/2/console

but now I also see http://www.couchbase.com/issues/browse/NCBC-545 on .net client 2.0

could you pay attention for the issue because it can solve (unsolved :) )problem SDKQE-539 - Memory issues on Jenkins slave is causing job to abort intermittently

Comment by Jeff Morris [ 10/Jul/14 ]
Andrei

Very good! Were you able to get the log file? I need that to ascertain if much hunch (and thus the patch) are correct.

Also, for those other tickets, please update those tickets instead of referencing them on this one :)

-Jeff




[NCBC-545] couchApiBase in pools/default/buckets/Bucket is in format http://IP:8092/BUCKET%2B/bucket_uuid( bucket_uuid is not required) Created: 08/Jul/14  Updated: 21/Aug/14  Resolved: 11/Jul/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2, 1.3.7
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-918

Attachments: Zip Archive Common.Logging.Log4Net.zip     Text File log.txt     File NCBC-545.logs    
Issue Links:
Relates to
relates to NCBC-550 Thread A is disposing of Bucket while... Resolved

 Description   
it's a blocker issue for at least couchbase-net-client against 3.0.0

for example, in the client when we build URL for queries we get uri like

http://10.3.4.144:8092/default+83794f1eaa6682ae0ada19cf8befd7ea/_design/test_design/_view/test_view/

3.0.0-918:

http://10.3.4.144:8091/pools/default/buckets/default
"couchApiBase":"http://10.3.4.144:8092/default%2B83794f1eaa6682ae0ada19cf8befd7ea"

2.5.0
http://10.3.4.147:8091/pools/default/buckets/default
"couchApiBase":"http://10.3.4.147:8092/default"


 Comments   
Comment by Aleksey Kondratenko [ 08/Jul/14 ]
Please explain _exactly_ how we're not allowed to add bucket uuid.

What actually breaks because of that and how exactly ?
Comment by Andrei Baranouski [ 08/Jul/14 ]
https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/DefaultHttpClient.cs#L202

to query client uses Uri(client.BaseAddress + "/" + path + queryString);

so, we get request like http://10.3.4.149:8092/default+58d6b67e4c80af5721e9ffcb72be97e1/_design/_design/test_design/_view/test_view that is not valid

[74.44 INFO] (SDKD log:137) [Couchbase.CouchbaseViewHandler|Warn] Design document not found, body: {"error":"not_found","reason":"missing"}
Comment by Aleksey Kondratenko [ 08/Jul/14 ]
Didn't get. Why _design gets duplicated? How existence of bucket uuid in couchApiBase (which is supposed to be to opaque) can affect it ?
Comment by Andrei Baranouski [ 08/Jul/14 ]
Thanks Alk for your comments

sorry, "_design" is my copy-paste error.

logs that I've added during debug

[39.66 INFO] (SDKD log:137) !!!!!!!!!!!!!!!!!!!!!!http://10.3.4.146:8092/default+83794f1eaa6682ae0ada19cf8befd7ea/_design/test_design/_view/test_view?

--- a/src/Couchbase/DefaultHttpClient.cs
+++ b/src/Couchbase/DefaultHttpClient.cs
@@ -105,6 +105,7 @@ namespace Couchbase
             public DefaultHttpResponseWrapper(HttpWebRequest request)
             {
                 this.request = request;
+ Console.WriteLine("!!!!!!!!!!!!!!!!!!!!!!" + request.Address);
             }


I figured out what the reason

http://10.3.4.146:8092/default+83794f1eaa6682ae0ada19cf8befd7ea/_design/test_design/_view/test_view - doesn't work, @Jeff seems like client "%2B" shouldn't decode "plus" symbol?
http://10.3.4.146:8092/default%2B83794f1eaa6682ae0ada19cf8befd7ea/_design/test_design/_view/test_view - works fine

not CB server issue
Comment by Wayne Siu [ 08/Jul/14 ]
Moving the ticket to NCBC project.
Comment by Jeff Morris [ 08/Jul/14 ]
What .NET client version is this impacting? 1.X or 2.X? Can you update the "Affects Version" in the ticket header?

Thanks,

Jeff
Comment by Wayne Siu [ 08/Jul/14 ]
Andrei,
I think it's 1.3. But please confirm. Thanks.
Comment by Andrei Baranouski [ 09/Jul/14 ]
it's 1.3.*
I used master branch with the latest commit
https://github.com/couchbase/couchbase-net-client/commit/fae1e56154cb069a598dbb43a9a3cbc2c284ff40
Comment by Jeff Morris [ 10/Jul/14 ]
Adding the bucket uuid is an unexpected change, however it appears that once the client is aware of it, it works the same as just using the bucket now.
Comment by Jeff Morris [ 10/Jul/14 ]
commit e5ead27c1efa5d33bb4d0ba9f55d424ffd9d3c79
Comment by Andrei Baranouski [ 10/Jul/14 ]
run with this commit http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/4/console


[42.98 INFO] (SDKD log:137)  at System.Collections.Generic.Dictionary`2.FindEntry(TKey key)
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Couchbase.Core.Server.GetBaseViewUri(String bucketName) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Server.cs:line 107
   at Couchbase.Core.Server.Send[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Server.cs:line 85
   at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-csharp-client-ng\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 481
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 122
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 225
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 191
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 123
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Comment by Jeff Morris [ 10/Jul/14 ]
Hi Andrei -

Can you enable logging on the SDK (instructions are here: https://www.couchbase.com/issues/browse/NCBC-536) and attach the log file to this ticket? Here is blog on how to set it up (use the app.config i attached to NCBC-536): http://blog.couchbase.com/couchbase-net-sdk-20-development-series-logging

What your providing me is the SDKQE log, what I need is the internal SDK log - it will give me a trace of what is happening before the exception occurs.

Thanks,

Jeff
Comment by Andrei Baranouski [ 11/Jul/14 ]
Hi Jeff,

I'm able to get only client's logs in console http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/20/console
will they be useful for you?

because when I try to use app.config with "log4net" based on your example https://gist.github.com/andreibaranouski/f0d357a0c5a5897d848a I get https://friendpaste.com/3bWmWMkzZ93yJMq5htPSMI
[37.00 INFO] (SDKD log:137) Unhandled Exception: System.TypeInitializationException: The type initializer for 'Couchbase.Core.ClusterManager' threw an exception. ---> Common.Logging.ConfigurationExcep
tion: Failed obtaining configuration for Common.Logging from configuration section 'common/logging'. ---> System.Configuration.ConfigurationErrorsException: An error occurred creating the configuratio
n section handler for common/logging: Unable to create type 'Common.Logging.Log4Net.Log4NetLoggerFactoryAdapter, Common.Logging.Log4Net' (C:\temp\sdkd-out-debug\SdkdConsole.exe.Config line 42) ---> Co
mmon.Logging.ConfigurationException: Unable to create type 'Common.Logging.Log4Net.Log4NetLoggerFactoryAdapter, Common.Logging.Log4Net' ---> System.IO.FileNotFoundException: Could not load file or ass
embly 'Common.Logging.Log4Net' or one of its dependencies. The system cannot find the file specified. ---> System.IO.FileNotFoundException: Could not load file or assembly 'file:///C:\Users\Administra
tor\Downloads\Couchbase-Net-Client-1.1.6\Common.Logging.Log4Net.dll' or one of its dependencies. The system cannot find the file specified.
[37.02 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[37.02 INFO] (SDKD log:137) at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, I
ntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolea
n throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Bo
olean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppre
ssSecurityChecks, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at SdkdConsole.Program.GetDependency(Object sender, ResolveEventArgs args) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Program.cs:line 107

do you know what is wrong here?


Comment by Jeff Morris [ 11/Jul/14 ]
Is this from the 1.3.X client or from the 2.X? Each version has their own logging infrastructure, what I attached in the app.config was specifically for 2.X.

Make sure you have the following files in your bin (i'll attach them):
-Common.Logging
-Common.Logging.Log4Net
-log4net

Note that the tester project is all set up for logging as an example.

Also, "file:///C:\Users\Administrator\Downloads\Couchbase-Net-Client-1.1.6\" looks like a very old client version? 1.1.6?


Comment by Andrei Baranouski [ 11/Jul/14 ]
these logs are from 2.x

yes, I also noticed about Couchbase-Net-Client-1.1.6. will try to figure out where does it come
Comment by Andrei Baranouski [ 11/Jul/14 ]
Couchbase-Net-Client-1.1.6 is from https://github.com/couchbase/sdkd-net/blob/client-2.0/src/SdkdConsole/Program.cs#L92 but it's not used (folder doesn't exist)
really, Common.Logging.Log4Net.dll log4net.dll have been lost in the project
I will modify sdkd-net project to add this libraries in references or replace hard code "C:\Users\Administrator\Downloads\Couchbase-Net-Client-1.1.6\" on c:\tmp\libraries(that will have missed libraries, for example for debug)

anyway, I upload log4net logs to the ticket from the latest 2.0 master branch
Comment by Andrei Baranouski [ 11/Jul/14 ]
log4net on 2.x client
Comment by Jeff Morris [ 11/Jul/14 ]
Great Andrei, thanks!
Comment by Jeff Morris [ 11/Jul/14 ]
Andrei -

After looking at the logs, I am going to close this ticket and open another (NCBC-550) - the error you are seeing in unrelated to this ticket.

-Jeff





[NCBC-567] Couchbase.IO.Strategies.DefaultIOStrategy - Could not authenticate default using Couchbase.Authentication.SASL.CramMd5Mechanism. Created: 28/Jul/14  Updated: 21/Aug/14  Resolved: 20/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 Beta

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive log.zip     Zip Archive NCBC-567.log.zip    

 Description   
after some time the tests begin to fall with
2014-07-28 00:59:20,864 [27] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Could not authenticate default using Couchbase.Authentication.SASL.CramMd5Mechanism.


Running Rb2Out-HYBRID: 2014-07-28 00:05:18
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-973\Rb2Out-HYBRID\07-28-14\025518\7d8cf558a26ca99fe4dc3028bc6b2ec0

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
        --rebalance-mode out --workload HybridWorkloadGroup
 \
        --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \
        --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \
        8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \
        --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1.47 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1.87 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1.88 INFO] (RunContext run:124) Starting cluster and driver
[1.89 INFO] (CBCluster startCluster:360) Node http://10.3.121.134:8091 is master now
[1.89 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe' 
[1.89 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2.26 INFO] (SDKD log:137) --- Logging Self-Test ---
[2.27 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message 
[2.27 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message 
[2.28 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message 
[2.28 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message 
[2.28 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675 
[2.42 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2.43 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[2.76 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.17929"}}
[4.02 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.135:8091,ns_1@10.3.121.135>
[4.06 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.136:8091,ns_1@10.3.121.136>
[4.11 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.3.206:8091,ns_1@10.3.3.206>
[4.18 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.18 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[4.70 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.134:8091
[4.71 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.3.206:8091
[4.71 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.135:8091
[4.71 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.136:8091
[4.73 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.134
[4.73 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.3.206
[4.73 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.136
[4.73 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.135
[4.94 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@641a034d
[4.94 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@19c6866d
[4.95 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@51c293f8
[4.95 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@5a9b8ff9
[4.95 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@6e0f0ad
[18.52 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.135:8091
[26.45 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.136:8091
[30.42 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.3.206:8091
[35.78 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[35.84 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[36.84 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[36.85 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[36.85 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[36.87 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[42.18 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[42.23 INFO] (RunContext run:143) Driver and cluster initialized
[42.26 INFO] (RunContext call:167) Running scenario..
[42.26 INFO] (Scenario run:72) Starting RAMP phase
[42.26 INFO] (Workload setupDesign:63) Creating design test_design
[42.51 INFO] (Workload setupDesign:80) Design creation done
[42.51 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[42.52 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[42.53 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101 
[42.54 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[42.56 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net 
[42.56 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e 
[42.59 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net 
[42.59 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821 
[42.91 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[42.91 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[43.02 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409 
[43.39 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {
[43.39 INFO] (SDKD log:137)  "0": 44,
[43.40 INFO] (SDKD log:137)  "264": 956
[43.40 INFO] (SDKD log:137) } 
[43.40 DEBUG] (Handle receiveMessage:158) < CB_VIEW_LOAD@101.2 => {[OK]: 44, [MEMD:GENERIC]: 956}
[43.40 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[43.40 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.41 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.41 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102 
[43.41 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.42 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[43.42 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@102.5 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}, Schema={InflateContent=meh, InflateLevel=40}, Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[43.82 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {
[43.82 INFO] (SDKD log:137)  "0": 39,
[43.83 INFO] (SDKD log:137)  "264": 961
[43.83 INFO] (SDKD log:137) } 
[43.83 DEBUG] (Handle receiveMessage:158) < CB_VIEW_LOAD@102.5 => {[OK]: 39, [MEMD:GENERIC]: 961}
[43.83 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[43.83 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.83 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.83 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103 
[43.83 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[43.85 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@103.8 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[43.85 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.85 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.86 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104 
[43.86 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.87 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.9
[43.88 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@104.10 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[43.88 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.88 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.88 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105 
[43.88 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.90 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.11
[43.90 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@105.12 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[43.90 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.90 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.13 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.91 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106 
[43.91 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.92 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.13
[43.93 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@106.14 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[43.93 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.93 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107 
[43.94 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.97 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.15
[43.97 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@107.16 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test_view}
[43.97 INFO] (Workload call:112) Preloading items into cluster before workload
[43.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[43.97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.17 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[43.98 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108 
[43.98 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[43.99 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.17
[43.99 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.18 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=false, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=0, DelayMin=0, IterWait=100, TimeRes=0}}
[51.93 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@108.18 => {[OK]: 364, [MEMD:GENERIC]: 14636}
[51.93 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@108.19
[51.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.22 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.23 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.26 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.27 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.21 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.20 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.25 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.24 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.94 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.28 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.95 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.29 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[51.96 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.96 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[51.99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.00 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.22
[52.00 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.00 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.00 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@111.30 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.00 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111 
[52.00 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110 
[52.01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115 
[52.01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109 
[52.01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114 
[52.01 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.31 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116 
[52.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117 
[52.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118 
[52.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112 
[52.02 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113 
[52.03 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.03 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.21
[52.03 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119 
[52.03 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.03 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@110.32 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.04 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.33 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.04 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.05 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120 
[52.07 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.07 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@115.26
[52.07 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@115.34 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.08 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.35 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.09 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121 
[52.10 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.10 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.20
[52.10 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@109.36 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.11 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.11 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@122.37 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.12 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 122 
[52.12 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.13 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@114.25
[52.13 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@114.38 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.13 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@123.39 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.13 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.14 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 123 
[52.16 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.16 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@116.27
[52.16 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@116.40 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.17 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@124.41 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.17 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.18 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 124 
[52.24 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.25 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@117.28
[52.25 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@117.42 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.26 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@125.43 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.26 INFO] (SDKD log:137) 
[52.26 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.26 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 125 
[52.27 INFO] (SDKD log:137) Unhandled Exception: System.Security.Authentication.AuthenticationException: default
[52.30 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.30 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@118.29
[52.31 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[52.31 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@118.44 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.31 INFO] (SDKD log:137)  at Couchbase.IO.Strategies.DefaultIOStrategy.Authenticate(IConnection connection) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 75
   at Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 40
   at Couchbase.Core.Server.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Server.cs:line 82
   at Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 121
   at Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\SetCommand.cs:line 77
   at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 191
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 123
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[52.31 INFO] (SDKD log:137) 
[52.32 INFO] (SDKD log:137) Unhandled Exception: System.Security.Authentication.AuthenticationException: default
[52.32 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@126.45 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.32 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.33 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 126 
[52.34 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.35 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.23
[52.35 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@112.46 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.37 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@127.47 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.37 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.37 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 127 
[52.40 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@113.24
[52.40 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@113.48 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.40 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.41 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@128.49 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[52.41 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle 
[52.42 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 128 
[52.45 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.45 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@119.31
[52.46 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@119.50 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.52 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.53 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@120.33
[52.53 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@120.51 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.55 INFO] (SDKD log:137) +++ Received exception of ID 0
[52.55 INFO] (SDKD log:137) 
[52.56 INFO] (SDKD log:137) Unhandled Exception: System.Security.Authentication.AuthenticationException: default
[52.63 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.63 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@121.35
[52.63 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@121.52 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.67 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.68 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@122.37
[52.68 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@122.53 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[52.76 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[52.77 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@123.39
[52.77 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@123.54 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[53.05 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[53.05 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@124.41
[53.05 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@124.55 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[53.15 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew 
[53.15 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@125.43
[53.16 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@125.56 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}}
[53.17 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3
[53.18 INFO] (SDKD log:137)  at Couchbase.IO.Strategies.DefaultIOStrategy.Authenticate(IConnection connection) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 75
   at Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:line 40
   at Couchbase.Core.Server.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Server.cs:line 82
   at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 282
   at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\GetCommand.cs:line 18
   at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 191
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 123
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[53.18 INFO] (SDKD log:137) 
[53.19 INFO] (SDKD log:137) Unhandled Exception: Couchbase.Configuration.ConfigException: Could not bootstrap default. See log for details.
[65.28 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[65.29 INFO] (SDKD log:137)  at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 210
   at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 82
   at Sdkd.Couchbase.ClientFactory.FindExistingClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 55
   at Sdkd.Couchbase.ClientFactory.CreateClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 262
   at SdkdConsole.Daemon.Handle.CreateHandle(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 84
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 103
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[65.30 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
[65.31 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.57
[65.31 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]

 Comments   
Comment by Jeff Morris [ 28/Jul/14 ]
This should be covered by NCBC-550.
Comment by Jeff Morris [ 28/Jul/14 ]
Should be covered by NCBC-550: commit 9ce8806edc38503086af1fe9af66484b5766859d
Comment by Andrei Baranouski [ 29/Jul/14 ]
Hi Jeff, still see the same issue for latest master branch:

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/44/consoleFull


client logs: http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/44/artifact/log.txt

2014-07-29 01:32:12,669 [32] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Could not authenticate default using Couchbase.Authentication.SASL.CramMd5Mechanism - 22da11dd-6230-4ead-88f8-b68476731dc0.
2014-07-29 01:32:12,667 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210
2014-07-29 01:32:12,670 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210
Comment by Jeff Morris [ 29/Jul/14 ]
The authentication error is happening because the CouchbaseCluster object is being finalizing (your leaking it somewhere in SDKQE most likely) and then trying do operations against it, but internally everything is disposed.

Line 200: 2014-07-29 01:32:06,326 [2] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Finalizing DefaultIOStrategy for 10.3.121.134:11210 - b76ca56e-e63e-413e-83d6-af0930c69b6e
Line 205: 2014-07-29 01:32:06,328 [2] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210
Line 246: 2014-07-29 01:32:06,335 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210
Line 251: 2014-07-29 01:32:06,336 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210
Line 252: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210
Line 253: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210
Line 255: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210
Line 258: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210
Line 260: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210
Line 261: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210
Line 262: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210
Line 263: 2014-07-29 01:32:06,337 [2] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210
Line 264: 2014-07-29 01:32:06,338 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
Line 265: 2014-07-29 01:32:06,341 [2] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
Line 271: 2014-07-29 01:32:06,342 [2] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
Line 272: 2014-07-29 01:32:06,403 [2] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
Line 273: 2014-07-29 01:32:06,404 [2] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - 9adbed7c-6e2a-4b6c-be4a-76897a4987bc

Thread [2] is the finalizer thread. Line 200 is suspect and likely a leak internally within the client, i believe it's fixed in a later commit but will dig deeper. Line 264 looks like the CouchbaseCluster object is no longer be referenced (leaked likely by SDKQE?) and is being finalized.

Let me look a little deeper - i am also going to change the log message so that when a TCP error is detected during auth, the log message states that as the reason.
Comment by Jeff Morris [ 29/Jul/14 ]
I think I found a problem with sdkq-net:

https://github.com/couchbase/sdkd-net/blob/client-2.0/src/Sdkd/Couchbase/ClientFactory.cs#L269

After calling ReleaseClient, which simply returns if the SharedClients is false, the CouchbaseCluster object is set to null:

https://github.com/couchbase/sdkd-net/blob/client-2.0/src/SdkdConsole/Daemon/Handle.cs#L68

The CouchbaseCluster object will be placed on the finalization queue (since it has a finalizer) and the instance will live on until GC kicks in and determines that it's finalizer must be called - this why we see:

Line 264: 2014-07-29 01:32:06,338 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster

In the logs. If the intention is to keep the CouchbaseClietnt alive through the entire run of tests, then we need to remove the SdkHandle = null. If the intention is to destroy the object, then call Dispose() then set to null and recreate the object when needed.

Comment by Andrei Baranouski [ 30/Jul/14 ]
Hi Jeff,

thanks for your research. I have fixed this sdkd defect before, but it did not help. so I decided that it did not affect the final result and I referred to Jenkins without fix( commited it only today in repo https://github.com/couchbase/sdkd-net/commit/1c505a352e114f4a36bd9c17a14b3cb66f0abb86)

my some thoughts:

I've added public method to get _disposed at runtime

$ git diff
diff --git a/Src/Couchbase/Core/Buckets/CouchbaseBucket.cs b/Src/Couchbase/Core/Buckets/CouchbaseBucket.cs
index d40ade3..978d32e 100644
--- a/Src/Couchbase/Core/Buckets/CouchbaseBucket.cs
+++ b/Src/Couchbase/Core/Buckets/CouchbaseBucket.cs
@@ -616,6 +616,12 @@ namespace Couchbase.Core.Buckets
            }
         }

+
+ public bool isDisposed()
+ {
+ return _disposed;
+ }


then I run the tests with changes like:

diff --git a/src/SdkdConsole/Daemon/Handle.cs b/src/SdkdConsole/Daemon/Handle.cs
index c76b291..76ef19c 100644
--- a/src/SdkdConsole/Daemon/Handle.cs
+++ b/src/SdkdConsole/Daemon/Handle.cs

@@ -187,7 +188,14 @@ public class Handle : ProtocolIOHandler
                        CurrentCommand = cmd;
                }

- cmd.Run();
+ try^M
+ {^M
+ cmd.Run();^M
+ } catch (System.Security.Authentication.AuthenticationException exc){^M
+ Console.WriteLine("!!!!!!!!!!!!!!!!System.Security.Authentication.AuthenticationException " + SdkHandle.isDisposed());^M
+ cmd.Run();^M
+^M
+ }^M


so, I only catch AuthenticationException , check _disposed status and rerun

with such changes/retry the tests run without this issue on client side
Comment by Jeff Morris [ 30/Jul/14 ]
Hi Andrei -

It definitely makes a difference with/respect to the state of client object. It's also possible for one error to hide the actual cause - in this case whenever authentication fails (it could be a leaked dead connection or a real auth failure or perhaps the node that the client attempted to auth against has been failed over and removed) the exception raised is "Could not authenticate default using Couchbase.Authentication.SASL.CramMd5Mechanism".

Once again, you cannot "null" out an object with a finalizer and expect it to work as expected (finalization is a complex topic, but it actually defers the destruction of the object for some time so any reference to it may be in use). You _must_ dispose of every client object you create - if not, then your not using the object correctly.

To make it easier to diagnose, i created a new ticket: https://www.couchbase.com/issues/browse/NCBC-571

This will ensure that when auth fails because of a network or TCP issue, the correct response will be returned.

I think we can close this ticket, ok?

Thanks,

Jeff

Comment by Jeff Morris [ 30/Jul/14 ]
Andrei -

This line is suspect: https://github.com/couchbase/sdkd-net/blob/1c505a352e114f4a36bd9c17a14b3cb66f0abb86/src/Sdkd/Couchbase/ClientFactory.cs#L76

You create a CouchaseCluster, but don't manage the reference to it, so it goes out of scope and eventually will be finalized. You should be caching this somewhere and then calling it's Dispose() method when you want to destroy the reference to it.

Look at this way, a CouchbaseCluster object manages CouchbaseBuckets and _all_ resources associated with the CouchbaseBucket object. If the CouchbaseCluster object goes out of scope and is not finalized, eventually it's finalizer will run - this could happen at any time, perhaps in the middle of a test. When it's finalizer is run, any buckets it's managing will be Disposed.

Thanks,

Jeff
Comment by Andrei Baranouski [ 31/Jul/14 ]
Hi Jeff,

I realized the main problem and made some changes localy to track refCount for CouchbaseBucket connections(for ShareClients cases) like:

            if (Clients.ContainsKey(key) && Clients[key].RefCount>0)
            {
                _Logger.Debug("Returning existing client for key '{0}'", key);
                clinfo = Clients[key];
                    clinfo.RefCount++;

                    Console.WriteLine(" num of RefCount on client side:" + clinfo.clientBucket.AddRef());
                    Console.WriteLine(" num of RefCount on SDKD:" + clinfo.RefCount);
                
                return clinfo.clientBucket;
            }

and in my runs I see that refCount is synced on client and sdkd side but problem still exists

I tried to isolate the problem and what I got:

I have only one working thread that run code(Upsert in loop):

 public class ViewLoader
    {
        // This class gets the view load stuff up and running..
        public Response Run(CouchbaseBucket cli, Request req, IDataSet ds)
        {
            JObject schema = (JObject)
                req.Payload[StringConst.V_SCHEMA];

            int i = 0;

            ResultSummary summary = new ResultSummary();

            var logger = Logging.Logging.GetLogger("ViewLoader");
            
            foreach (KeyValuePair<string, string> kvp in ds)
            {
                i++;
                Console.WriteLine(i);
                // Modify the schema..
                schema[StringConst.V_KIDENT] = kvp.Key;
                schema[StringConst.V_KSEQ] = i;

                string jstr = schema.ToString(Formatting.None);
                // This is what we should use. But the SDK is buggy pending some gerrit commits..
#if OBSERVE_OK
var res = cli.ExecuteStore(StoreMode.Set, kvp.Key, jstr, PersistTo.One);
#endif
                logger.Info(kvp.Key);
                var res = cli.Upsert(kvp.Key, jstr);
                logger.Info(res.Status);
                logger.Info(res.Success);
                logger.Info(res.Message);
                summary.Submit(res);
                System.Threading.Thread.Sleep(3000);
            }
            Console.WriteLine("Done +" + i +" Operations");
.....



C:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,14 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to C:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1060\Rb2Out-HYBRID\07-31-14\049533\51a5f64d8fb40eef389a8779bf9167b3

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1,42 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1,70 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,71 INFO] (RunContext run:124) Starting cluster and driver
[1,71 INFO] (CBCluster startCluster:360) Node http://10.3.121.134:8091 is master now
[1,71 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,71 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[1,90 INFO] (SDKD log:137) --- Logging Self-Test ---
[1,91 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[1,91 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[1,91 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[1,91 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[1,91 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,27 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,27 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,44 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[7,60 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.135:8091,ns_1@10.3.121.135>
[8,12 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.136:8091,ns_1@10.3.121.136>
[8,61 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.3.206:8091,ns_1@10.3.3.206>
[9,13 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[9,13 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[11,15 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.136:8091
[11,15 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.135:8091
[11,15 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.134:8091
[11,16 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.3.206:8091
[11,17 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.3.206
[11,17 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.136
[11,18 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.134
[11,18 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.135
[12,02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@5c921914
[12,02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@21d55d9
[12,02 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@40439a1f
[12,03 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1eed1f10
[12,03 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@16d96b45
[25,03 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.135:8091
[31,46 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.136:8091
[40,68 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.3.206:8091
[51,74 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[52,73 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[52,73 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[52,98 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[53,23 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[61,97 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[64,85 INFO] (RunContext run:143) Driver and cluster initialized
[65,11 INFO] (RunContext call:167) Running scenario..
[65,11 INFO] (Scenario run:72) Starting RAMP phase
[65,11 INFO] (Workload setupDesign:63) Creating design test_design
[66,40 INFO] (Workload setupDesign:80) Design creation done
[66,40 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[66,40 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[66,41 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[66,42 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[66,44 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[66,44 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[66,47 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[66,47 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[70,53 INFO] (SDKD log:137) default
[70,53 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[70,54 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}, Schema={InflateContent=meh, InflateLevel=40}, O
ptions={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[70,54 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[70,54 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[70,54 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[70,55 INFO] (SDKD log:137) "DS": {
[70,55 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed",
[70,55 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed",
[70,55 INFO] (SDKD log:137) "Count": 1000,
[70,55 INFO] (SDKD log:137) "Continuous": false,
[70,55 INFO] (SDKD log:137) "VSize": 512,
[70,55 INFO] (SDKD log:137) "Repeat": "rep",
[70,56 INFO] (SDKD log:137) "KSize": 12
[70,56 INFO] (SDKD log:137) },
[70,56 INFO] (SDKD log:137) "Schema": {
[70,56 INFO] (SDKD log:137) "InflateContent": "meh",
[70,56 INFO] (SDKD log:137) "InflateLevel": 40
[70,56 INFO] (SDKD log:137) },
[70,56 INFO] (SDKD log:137) "Options": {
[70,56 INFO] (SDKD log:137) "DelayMax": 0,
[70,57 INFO] (SDKD log:137) "DelayMin": 0,
[70,57 INFO] (SDKD log:137) "IterWait": 1,
[70,57 INFO] (SDKD log:137) "TimeRes": 0
[70,57 INFO] (SDKD log:137) }
[70,57 INFO] (SDKD log:137) }
[70,57 INFO] (SDKD log:137) 1
[70,57 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep0rep0rep0
[71,51 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[71,51 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[71,51 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[74,51 INFO] (SDKD log:137) 2
[74,52 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep1rep1rep1
[75,48 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[75,48 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[75,48 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[78,48 INFO] (SDKD log:137) 3
[78,48 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep2rep2rep2
[79,43 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[79,43 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[79,43 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[82,43 INFO] (SDKD log:137) 4
[82,43 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep3rep3rep3
[83,45 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[83,45 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[83,45 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[86,45 INFO] (SDKD log:137) 5
[86,45 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep4rep4rep4
[86,69 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[86,69 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[86,69 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[89,69 INFO] (SDKD log:137) 6
[89,69 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep5rep5rep5
[89,96 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[89,96 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[89,96 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[92,96 INFO] (SDKD log:137) 7
[92,96 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep6rep6rep6
[93,20 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[93,20 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[93,20 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[96,20 INFO] (SDKD log:137) 8
[96,20 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep7rep7rep7
[96,44 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[96,44 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[96,44 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[99,44 INFO] (SDKD log:137) 9
[99,44 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep8rep8rep8
[99,91 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[99,91 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[99,91 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[102,91 INFO] (SDKD log:137) 10
[102,91 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep9rep9rep9
[103,15 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[103,15 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[103,15 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[106,15 INFO] (SDKD log:137) 11
[106,15 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep10rep10rep10
[106,39 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[106,39 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[106,39 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[109,39 INFO] (SDKD log:137) 12
[109,39 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep11rep11rep11
[109,83 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[109,83 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[109,83 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[112,83 INFO] (SDKD log:137) 13
[112,83 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep12rep12rep12
[113,07 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[113,07 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[113,07 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[116,07 INFO] (SDKD log:137) 14
[116,07 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep13rep13rep13
[116,31 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[116,31 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[116,31 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[119,31 INFO] (SDKD log:137) 15
[119,31 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep14rep14rep14
[119,54 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[119,55 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[119,55 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[122,55 INFO] (SDKD log:137) 16
[122,55 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep15rep15rep15
[122,80 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[122,80 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[122,80 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[125,80 INFO] (SDKD log:137) 17
[125,80 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep16rep16rep16
[126,03 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[126,04 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[126,04 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[129,03 INFO] (SDKD log:137) 18
[129,04 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep17rep17rep17
[129,27 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Success
[129,27 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] True
[129,27 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info]
[132,27 INFO] (SDKD log:137) 19
[132,28 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep18rep18rep18
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ClientFailure
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] False
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Cannot access a disposed object
[132,29 INFO] (SDKD log:137) Имя объекта: "System.Net.Sockets.NetworkStream".
[132,29 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[133,29 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[133,30 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60]
[133,30 DEBUG] (Handle sendMessageAsync:183) > GOODBYE@0.4
[133,30 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
        at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60]
        at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60]
        at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60]
        at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:121) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]
[133,40 ERROR] (BRun main:507) Caught exception
java.io.IOException: java.sql.SQLException: Connection has already been closed
        at com.couchbase.sdkdclient.batch.BRun.uploadFile(BRun.java:145) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:284) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
Caused by: java.sql.SQLException: Connection has already been closed
        at com.j256.ormlite.jdbc.JdbcConnectionSource.getReadWriteConnection(JdbcConnectionSource.java:177) ~[ormlite-jdbc-4.47.jar:na]
        at com.j256.ormlite.dao.BaseDaoImpl.create(BaseDaoImpl.java:306) ~[ormlite-core-4.47.jar:na]
        at com.couchbase.sdkdclient.batch.history.HistoryDB.addLog(HistoryDB.java:41) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        at com.couchbase.sdkdclient.batch.BRun.uploadFile(BRun.java:143) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
        ... 4 common frames omitted
Завершить выполнение пакетного файла [Y(да)/N(нет)]? Y

regardless of the timeout System.Threading.Thread.Sleep(3000) test always failed after 20 iterations
in client logs after 20 iteraions we always get "Disposing ConnectionPool for 10.3.121.134:11210" and all other operations failed with Cannot access a disposed object ....

please also note that after the first failure status code is 409:

[132,28 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ViewFillerSeedrep18rep18rep18
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] ClientFailure
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] False
[132,29 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Cannot access a disposed object
[132,29 INFO] (SDKD log:137) Имя объекта: "System.Net.Sockets.NetworkStream".
[132,29 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409


client logs: https://friendpaste.com/2HsC31DsjLD4vA68rACp9D



Comment by Andrei Baranouski [ 04/Aug/14 ]
will try to rewrite sdkd-net using 2.0 approach to avoid

2014-08-01 03:45:18,498 [15] WARN Couchbase.IO.ConnectionBase - Opcode=Set | Key=SimpleKeyREP14831REP14831REP14831REP14831 | Host=10.3.121.136:11210
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 37
Comment by Andrei Baranouski [ 06/Aug/14 ]
with commit SHA: 940183971ea8d0570033ed28bba2e7c84ddc2b71
I still see the exception in different test phases but now tests are moving far beyond the initial load as described in https://www.couchbase.com/issues/browse/NCBC-567?focusedCommentId=96590&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-96590

System.Security.Authentication.AuthenticationException: default
[223,13 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Authenticate(IConnection connection) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 78
[223,13 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 43
[223,14 INFO] (SDKD log:137) в Couchbase.Core.Server.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Server.cs:строка 84
[223,14 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 96
[223,14 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 272
[223,15 INFO] (SDKD log:137) в Sdkd.Views.ViewLoader.Run(CouchbaseBucket cli, Request req, IDataSet ds) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewLoader.cs:строка 44
[223,15 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 193
[223,15 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 153
[223,15 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[223,15 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[223,15 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[223,16 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[223,16 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart()
Comment by Jeff Morris [ 06/Aug/14 ]
Andrei - can you include updated client logs?

Note that this authentication error will happen in the following cases:

   a) The bucket name and password are invalid
   b) The bucket does not exist! (server returns "Auth Error" when a bucket has been deleted or does not exist)
   c) When a TCP error happens on the connection - the server could be removed from the cluster or not reachable for example. In which case the message returned will have additional info associated with it (like exception that was caught)

Without the client logs it's nearly impossible to diagnose the cause.

-Jeff
Comment by Andrei Baranouski [ 12/Aug/14 ]
Hi Jeff,

please see client logs for test with AuthenticationException https://friendpaste.com/45q4G93yMzXR2SwSGxcCUA

System.Security.Authentication.AuthenticationException: default
[7860,06 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Authenticate(IConnection connection) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 78
[7860,06 INFO] (SDKD log:137) в Couchbase.IO.Strategies.DefaultIOStrategy.Execute[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\IO\Strategies\DefaultIOStrategy.cs:строка 43
[7860,06 INFO] (SDKD log:137) в Couchbase.Core.Server.Send[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Server.cs:строка 84
[7860,07 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 97
[7860,07 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:строка 310
[7860,07 INFO] (SDKD log:137) в Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:строка 77
[7860,08 INFO] (SDKD log:137) в Sdkd.Commands.KVCommand.Run() в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:строка 25
[7860,08 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 231
[7860,08 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:строка 153
[7860,08 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[7860,08 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[7860,09 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[7860,09 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)


 logs https://www.couchbase.com/issues/secure/attachment/21550/NCBC-567.log.zip

btw, I use.net client with the latest commit from gerrit http://review.couchbase.org/#/c/40351/
Comment by Jeff Morris [ 12/Aug/14 ]
From the logs:
Line 229804: 2014-08-12 18:37:23,661 [41] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: ccea0dab-a040-42d7-9d23-3d67a2c19499 | 10.3.4.147:11210 | [0, 1] - 81c963c0-cbb4-4c91-800f-6800b885c6e8 - Disposed: False
Line 229805: 2014-08-12 18:37:23,661 [41] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authenticating socket ccea0dab-a040-42d7-9d23-3d67a2c19499
Line 229837: 2014-08-12 18:37:24,381 [41] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authentication for socket ccea0dab-a040-42d7-9d23-3d67a2c19499 failed: Timed out.

My guess is that the config is stale and the node it's trying to auth against has been removed from the cluster. There is some other stuff going on as well:

2014-08-12 16:27:39,204 [4] WARN Couchbase.Core.ClusterManager - Couchbase.Configuration.ConfigException: Could not retrieve configuration for default. Reason: ?? ?  {"rev":69,"name":"default","uri":"/pools/default/buckets/default?bucket_uuid=0419447eedc8dc9da2702ed0e4d31f09","streamingUri":"/pools/default/bucketsStreaming/default?bucket_uuid=0419447eedc8dc9da2702ed0e4d31f09","nodes":[{"couchApiBase":"http://10.3.4.144:8092/default%2B0419447eedc8dc9da2702ed0e4d31f09","hostname":"10.3.4.144:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://10.3.4.145:8092/default%2B0419447eedc8dc9da2702ed0e4d31f09","hostname":"10.3.4.145:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://10.3.4.146:8092/default%2B0419447eedc8dc9da2702ed0e4d31f09","hostname":"10.3.4.146:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://10.3.4.147:8092/default%2B0419447eedc8dc9da2702ed0e4d31f09","hostname":"10.3.4.147:8091","ports":{"proxy":11211,"direct":11210}}],"nodesExt":[{"services":{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091},"hostname":"10.3.4.144"},{"services":{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091},"hostname":"10.3.4.145"},{"services":{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091},"hostname":"10.3.4.146"},{"services":{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091},"hostname":"10.3.4.147"}],"nodeLocator":"vbucket","uuid":"0419447eedc8dc9da2702ed0e4d31f09","ddocs":{"uri":"/pools/default/buckets/default/ddocs"},"vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.3.4.144:11210","10.3.4.145:11210","10.3.4.146:11210","10.3.4.147:11210"],"vBucketMap":[[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],Timed out.
   в Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:строка 67
   в Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:строка 159
2014-08-12 16:27:39,210 [4] DEBUG Couchbase.Core.ClusterManager - Trying to boostrap with Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.

Comment by Jeff Morris [ 20/Aug/14 ]
Fixed here: http://www.couchbase.com/issues/browse/NCBC-592




[NCBC-557] couchbaseBucket.CreateQuery(*).BucketName equals "" -> NullReferenceException in Queries Created: 23/Jul/14  Updated: 21/Aug/14  Resolved: 23/Jul/14

Status: Closed
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 2.0 DP2
Fix Version/s: 2.0 DP3

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
have a code like:
var query =couchbaseBucket.CreateQuery( true, dname, vname);
Console.WriteLine("couchbaseBucket.Name: " + couchbaseBucket.Name);
Console.WriteLine("bucket name in query: " + query.BucketName);
IViewResult<dynamic> view = couchbaseBucket.Query<dynamic>(query);

output:
couchbaseBucket.Name: default
bucket name in query:


so, query.BucketName is "" after CreateQuery

couchbaseBucket.Query<dynamic>(query); ====>


     public IViewResult<T> Send<T>(IViewQuery query)
        {
            var baseUri = GetBaseViewUri(query.BucketName);========>
            query.BaseUri(baseUri);
            return ViewClient.Execute<T>(query);
        }



        public string GetBaseViewUri(string bucketName)
        {
            var uri = _nodeInfo.CouchApiBase;
            var bucketConfig = _clientConfiguration.BucketConfigs[bucketName]; // !!!bucketName equals and exception appears:




System.NullReferenceException: Object reference not set to an instance of the object.
[93,80 INFO] (SDKD log:137) в Couchbase.Core.Server.GetBaseViewUri(String bucketName) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Server.cs:line109
[93,81 INFO] (SDKD log:137) в Couchbase.Core.Server.Send[T](IViewQuery query) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Server.cs:line 88
[93,81 INFO] (SDKD log:137) в Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) в c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line507
[93,81 INFO] (SDKD log:137) в Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line126
[93,82 INFO] (SDKD log:137) в Sdkd.Views.ViewCommand.Run() в c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line231
[93,82 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.DispatchCommand(Request req) в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line191
[93,82 INFO] (SDKD log:137) в SdkdConsole.Daemon.Handle.Run() в c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line123
[93,82 INFO] (SDKD log:137) в System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[93,83 INFO] (SDKD log:137) в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[93,83 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[93,83 INFO] (SDKD log:137) в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)



 Comments   
Comment by Jeff Morris [ 23/Jul/14 ]
Gerrit SHA: commit 9334ae491612644a306059aeb5d554c6132a22ae




[NCBC-615] [ssl] data load stuck with 'Socket Exception was unhandled' Created: 29/Aug/14  Updated: 02/Sep/14  Resolved: 30/Aug/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-615_1.png     PNG File NCBC-615_2.png     PNG File NCBC-615_3.png     Zip Archive NCBC-615.zip     File test.logs    

 Description   
Hi Jeff,

I don't see the issue without ssl but see it in maby tests with ssl.
I'm not 100% sure that it is the client problem, but I use the same sdkd code and there is no such problem.

in short, I see that not all the data has been downloaded, and tests hang. when I stop the command line to debug, I see the following exception. Could you take a look at the problem? my research have so far failed. I am ready to provide the environment or provide more detailed information if required.

 Comments   
Comment by Jeff Morris [ 29/Aug/14 ]
Can you post the stacktrace? Click on the "View Detail" in the exception dialog and navigate through. If an inner exception exists, make sure it's included.

Comment by Andrei Baranouski [ 30/Aug/14 ]
When I run with http://review.couchbase.org/#/c/41064/ (NCBC-610) then the problem goes away for this test




[NCBC-563] NullReferenceException in CouchbaseConfigContext.LoadConfig Created: 25/Jul/14  Updated: 30/Sep/14  Resolved: 15/Sep/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0 Beta

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive log.zip    

 Description   
very often in different tests I see the following exception


http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/38/consoleFull

Running SvcRestart-HYBRID: 2014-07-25 05:20:15
..............

[365.86 INFO] (SDKD log:137) Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
[366.20 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[366.20 INFO] (SDKD log:137)  at Couchbase.Configuration.CouchbaseConfigContext.LoadConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\CouchbaseConfigContext.cs:line 51
   at Couchbase.Configuration.ConfigContextBase..ctor(IBucketConfig bucketConfig, ClientConfiguration clientConfig, Func`2 ioStrategyFactory, Func`3 connectionPoolFactory, Func`5 saslFactory, IByteConverter converter, ITypeSerializer serializer) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 51
   at Couchbase.Configuration.CouchbaseConfigContext..ctor(IBucketConfig bucketConfig, ClientConfiguration clientConfig, Func`2 ioStrategyFactory, Func`3 connectionPoolFactory, Func`5 saslFactory, IByteConverter converter, ITypeSerializer serializer) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\CouchbaseConfigContext.cs:line 22
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(IBucketConfig bucketConfig) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 63
   at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.GetConfig(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\CarrierPublication\CarrierPublicationProvider.cs:line 44
   at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 155
   at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 82
   at Sdkd.Couchbase.ClientFactory.FindExistingClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 55
   at Sdkd.Couchbase.ClientFactory.CreateClient(ClientConfiguration config) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Couchbase\ClientFactory.cs:line 262
   at SdkdConsole.Daemon.Handle.CreateHandle(Request req) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 84
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 103
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[366.20 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
[366.21 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@203.259
[366.21 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]


client logs http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/38/artifact/log.txt

Jeff, believe that this is a problem on client side?



 Comments   
Comment by Jeff Morris [ 25/Jul/14 ]
Yes, I'll take a look into this today Andrei.
Comment by Jeff Morris [ 25/Jul/14 ]
Andrei - can you attach the SDK logs to this ticket?
Comment by Jeff Morris [ 25/Jul/14 ]
This is what's happening:

When the client attempts to bootstrap and the node it elects to bootstrap off of is in an inconsistent state, during the SASL process the first fails and thus the SaslMechanism is never chosen leading to a subsequent NRE:

https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Authentication/SASL/SaslFactory.cs#L24

https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase/Configuration/CouchbaseConfigContext.cs#L51

The simple solution seems to be check for null - which will push the problem further out. It should be a "cannot bootstrap error, please try again" perhaps looping through the bootstrap list in the configuration.
Comment by Jeff Morris [ 25/Jul/14 ]
commit b5fc255555f26eb5f691eb0441c7997577fc79e6




[NCBC-626] HttpRequestException in queries after rebalance out Created: 07/Sep/14  Updated: 30/Sep/14  Resolved: 25/Sep/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 DP3
Fix Version/s: 2.0.0

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-1208

Attachments: HTML File 7c866d0c3984e2387129b6f66f32f19f-CB.html     HTML File 7c866d0c3984e2387129b6f66f32f19f-HT.html     Zip Archive NCBC-626.log.zip     Zip Archive NCBC-626_new.log.zip     File NCBC-626_with_gerrit.7z     Zip Archive NCBC-626_without_ssl.log.zip     Text File test_logs_new.txt     Text File test.txt    

 Description   
c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,12 INFO] (BRun run:441) Initializing history database

============================================