[NCBC-655] .NET Client seeing many pauses during Rebalance Created: 23/Sep/14  Updated: 23/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 1.3.9
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I have been benchmarking the Java client vs .NET client. During a rebalance, I see frequent pauses within the .NET client that greatly impacts overall throughput. These pauses are not seen in the Java client with equivalent code (gets). I've captured a 30 second video as well as a 30 second network capture, attached. It is connecting to a 3 node 2.5.1 cluster where I add or remove a node, and rebalance.

ASK: Why is the .NET client pausing for 10-15 seconds each minute during a rebalance?

http://screencast.com/t/kNyXkauaEPK3

Here is the relevant C# code, performing a GetWithCas with random keys in a loop:

private static void Main(string[] args)
        {
         
            var config = new CouchbaseClientConfiguration();

            config.Bucket = "TestBucket";
            config.Username = "";
            config.Password = "";

            config.SocketPool.MaxPoolSize = 20;
            config.SocketPool.MinPoolSize = 20;
            config.RetryCount = 10;
            config.HeartbeatMonitor.Interval = 30000;
            config.SocketPool.ReceiveTimeout = TimeSpan.FromMilliseconds(2500);
            config.RetryTimeout = TimeSpan.FromMilliseconds(1000);

            config.Urls.Add(new Uri("http://10.4.2.57:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.107:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.108:8091/pools"));

            var Client = new CouchbaseClient(config);

            // create threads to read random documents:

            ParallelOptions parallelOptions = new ParallelOptions();
            parallelOptions.MaxDegreeOfParallelism = 1;
            Parallel.For((long)0, 1, parallelOptions, i =>
            {
                for (int j = 100071; j < 1000000; j++)
                {
                    string key = "Couch_" + j.ToString();
                    var casVal = Client.GetWithCas(key);
                    if (casVal.Result != null)
                    {
                        Console.WriteLine("j=" + j.ToString());
                    }
                }
            });

            Client.Dispose();
        }
    }


FYI, here is the corresponding Java code that does not paused during a rebalance:

public static void main(String[] args) throws Exception {
        new App().run();
    }
    
    private void run() throws InterruptedException, ExecutionException {
        int numThreads = 1;
        
        ExecutorService pool = Executors.newFixedThreadPool(numThreads);
        List<Future> fs = new ArrayList<Future>(numThreads);
        for (int i=0; i<numThreads; i++) {
            fs.add(pool.submit(new Reader(i+1, 5000000)));
        }
        for (Future f : fs) {
            f.get();
        }
    }

    private class Reader implements Runnable {
        
        private int id;
        private long runningTime;
        
        public Reader(int id, long runningTime) {
            this.id = id;
            this.runningTime = runningTime;
        }
        
        @Override
        public void run() {
            long n=100071;
            long start, end=0;
            long initTime = System.currentTimeMillis();
            List<URI> uris = null;
            try {
                uris = Arrays.asList(new URI("http://10.4.2.57:8091/pools"));
            } catch (URISyntaxException e) {
                
                e.printStackTrace();
            }
            String bucketName = "TestBucket";
            String result = "";
            String password = "";
            CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
            builder.setReadBufferSize(20000);
            builder.setOpTimeout(2500);
            builder.setFailureMode(FailureMode.Redistribute);
            builder. setOpQueueMaxBlockTime(1000);
            CouchbaseConnectionFactory connectionFactory = null;
            CouchbaseClient client = null;
            try {
                connectionFactory =
                builder.buildCouchbaseConnection(uris, bucketName, "", "");
                client = new com.couchbase.client.CouchbaseClient(connectionFactory);
            } catch (IOException e) {
                e.printStackTrace();
            }
            
            while (true) {
                String key = "Couch_" + n;
                start = System.currentTimeMillis();
                
                if (client.gets(key) !=null) {
                    System.out.println("N=" + n);
                }
                
                n++;
                if (runningTime>0 && (end-initTime)>runningTime) {
                    break;
                }
            }
        }
    }


 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
Jeff -

Thanks for doing this :) Not sure why the client pauses - I have some ideas, though; I'll have to take a deeper look.

-Jeff




[NCBC-654] Getting started guide for .NET is several versions behind the latest Created: 22/Sep/14  Updated: 22/Sep/14  Resolved: 22/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: docs
Affects Version/s: None
Fix Version/s: 1.3.9

Type: Task Priority: Major
Reporter: Ian McCloy Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
http://www.couchbase.com/communities/net/getting-started << Needs updating to 1.3.9

 Comments   
Comment by Jeff Morris [ 22/Sep/14 ]
Hi Ian -

I updated the links to 1.3.9. We do have a release process, I just missed this page...my comment in NCBC-389 explains why.

I'll make sure this gets updated with future releases.

Thanks,

Jeff




[NCBC-653] MemcachedBucket does not have SendWithRetry method Created: 20/Sep/14  Updated: 20/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[NCBC-652] Add support for bulk operations: Get and Upsert Created: 20/Sep/14  Updated: 21/Sep/14  Resolved: 21/Sep/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Task Priority: Blocker
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Add support for multi get/upsert.

 Comments   
Comment by Jeff Morris [ 21/Sep/14 ]
http://review.couchbase.org/#/c/41538/

Missing some unit tests which will be done a bit later as a separate commit.




[NCBC-651] Create Tests to ensure 1.X and 2.X work together in the same app domain Created: 20/Sep/14  Updated: 20/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
These tests should also cover all common flags integration concerns.




[NCBC-650] System.OperationCanceledException in CouchbaseBucket.Query when 3/4 nodes restarted Created: 19/Sep/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File test.txt    

 Description   
============================================================
Running SvcRestart-HYBRID: 2014-09-19 01:52:57
    service/restore_delay=5
    workload=HybridWorkloadGroup
    service/name=SYSV_SERVICE
    service/action=kill
    service/count=3
    testcase=ServiceFailureScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-19-14\031977\886b17542e83bb34cd392d356a0403e4

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --service-restore_delay 5 --workload HybridWorkloadGroup
 \ --service-name SYSV_SERVICE --service-action kill --service-count
 \ 3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
 \ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8


[1844.07 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.135
[1845.76 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@6059c26
[1846.78 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.136
[1848.51 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1b9e7c7b
[1849.59 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.3.206
[1851.20 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@60b8af93
[1852.21 INFO] (Scenario run:97) CHANGE phase done
[1852.21 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[1868.25 INFO] (SDKD log:137)
[1868.29 INFO] (SDKD log:137)
[1868.29 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception:
[1868.29 INFO] (SDKD log:137) Unhandled Exception:
[1868.29 INFO] (SDKD log:137)
[1868.33 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception:
[1868.33 INFO] (SDKD log:137) Unhandled Exception:
[1868.40 INFO] (SDKD log:137) Unhandled Exception: System.OperationCanceledException: The operation was canceled.
[1868.41 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[1868.41 INFO] (SDKD log:137) at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
   at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:lin
e 0
   at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[1868.41 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[1868.43 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3
[1868.43 INFO] (SDKD log:137) at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
   at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:lin
e 1141
   at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
   at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
   at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
   at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
   at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
[1868.44 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.



 Comments   
Comment by Andrei Baranouski [ 19/Sep/14 ]
https://s3.amazonaws.com/bugdb/jira/NCBC-650/4b913ca6/NCBC-650.zip
Comment by Jeff Morris [ 19/Sep/14 ]
This is a timeout on a view query, so it should be handled and returned in the IViewResult.Exception field; it shouldn't be an uncaught exception.




[NCBC-649] infinite loop 'Trying again' with 'No connections currently available' for ReAdd2 test when nodes returned to cluster Created: 19/Sep/14  Updated: 20/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-649_1.png     PNG File NCBC-649_2.png     PNG File NCBC-649_3.png     PNG File NCBC-649_4.png     File NCBC-649.7z     File NCBC-649_gerrit.7z     PNG File NCBC-649.png    

 Description   
run with commit https://github.com/couchbase/couchbase-net-client/commit/24714876f76207a5fc7d72e88bd1f9c57909fcdb

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/102/console

Running ReAdd2-HYBRID: 2014-09-18 23:50:57
    workload=HybridWorkloadGroup
    failover/next_action=FO_READD_REBALANCE
    failover/count=2
    failover/next_delay=45
    testcase=FailoverScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\ReAdd2-HYBRID\09-18-14\024657\8c847f52905fd0bf669803f16c2dc716

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
        --workload HybridWorkloadGroup --failover-next_action
 \
        FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
 \
        --testcase FailoverScenario -C share\rexec --rexec_path
 \
        C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \
        --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \
        --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \
        --cluster_ssh-username=root --cluster_ssh-password=couchbase


2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
2014-09-18 23:56:46,614 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 1 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,614 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#878 on thread 42
2014-09-18 23:56:46,614 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 609 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 - rev878
2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP4463REP4463REP4463REP4463REP4463 using server 10.3.121.134:11210
2014-09-18 23:56:46,615 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b | 10.3.121.134:11210 | [0, 2] - b20c68a7-f922-486e-af16-f880d88bfff6 - Disposed: False
2014-09-18 23:56:46,615 [42] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP4463REP4463REP4463REP4463REP4463 using 609 on 10.3.121.134:11210
2014-09-18 23:56:46,616 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b on 10.3.121.134:11210 - b20c68a7-f922-486e-af16-f880d88bfff6
2014-09-18 23:56:46,619 [42] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1736
2014-09-18 23:56:46,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:46,880 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
   at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
   at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 2 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 failed after 3 retries. Reason: VBucketBelongsToAnotherServer
2014-09-18 23:56:49,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:51,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:54,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:56,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:56:59,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:01,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-18 23:57:04,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


..................


2014-09-19 01:09:51,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:54,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:56,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:09:59,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
2014-09-19 01:10:01,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False


test stuck, not all items loaded
run was via jenkins, so I'm not able to get stacktraces



 Comments   
Comment by Jeff Morris [ 19/Sep/14 ]
Hmm, looks like connection pool exhausted, but none of the in-progress operations are timing out? Strange.
Comment by Andrei Baranouski [ 19/Sep/14 ]
right, something like that
Comment by Andrei Baranouski [ 20/Sep/14 ]
catched it in manual run


  [In a sleep, wait, or join]
  mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) + 0x22 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) + 0x28 bytes
  mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout) + 0xd bytes
> Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 123 + 0x28 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
 
        .................................
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Couchbase.IO.IConnectionPool.Acquire() Line 178 + 0x9 bytes C#
  Couchbase.dll!Couchbase.IO.Strategies.DefaultIOStrategy.Execute<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 40 + 0xc bytes C#
  Couchbase.dll!Couchbase.Core.Server.Send<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 88 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry<object>(Couchbase.IO.Operations.IOperation<object> operation) Line 103 + 0x4a bytes C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.Upsert<object>(string key, object value) Line 332 + 0x31 bytes C#
  Sdkd.dll!Sdkd.Commands.SetCommand.ExecuteSingle(string key, string value) Line 78 + 0x1e bytes C#
  Sdkd.dll!Sdkd.Commands.KVCommand.Run() Line 24 + 0x36 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.Buckets.CouchbaseBucket cli) Line 266 + 0x9 bytes C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 146 + 0xf bytes C#
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x6f bytes
  mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0xa7 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x16 bytes
  mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x41 bytes
  mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes
  [Native to Managed Transition]


please note, this run was with some commits from gerrit

$ git log
commit 8afa0ace0ac366e53dcc07a62c23ffd6604f6015
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 10:43:37 2014 -0700

    NCBC-639: ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query)

    Change-Id: I41bd3d93b31b98a19b8fb23d432713f5034debd7

commit 602181eed21614451ccaf0f848337f9c382ae578
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Fri Sep 12 00:10:53 2014 -0700

    NCBC-590: Refactor logging so that ODE are not raised when it references a disposed object

    Change-Id: I5d81dba00d64e4caf34d165491624e2c02a9949b

commit f9d4efb511c93a64b2a844e41849177da1f4442a
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:53:12 2014 -0700

    Refactor unit tests to reflect changes in exceptions when bootstrap fails

    Change-Id: I387aa3c8a0d8954102265e27dbbe4f6d84c4d104

commit 2a054aceab491ed338f1fc435d0a48eb4d31c43c
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 22:13:02 2014 -0700

    NCBC-624: Refactor View API

    From Michael's review:

    "One thing that I was confused about was the View query DSL. The required
    argument is development, but the view and design document name could be
    omitted by the DSL user since they are elements like limit and so forth. I
    think it would be better to turn this around. For example in Java there is
    ViewQuery.from('designDoc', 'view') and then the DSL starts.
    So you can never miss design doc and view name by accident. And I think
    you can go with a development = false default, since that's what people
    use most of the time. I think the main question is, if I miss adding
    design doc or view name by accident, does it blow up at runtime or at
    compile time?"

    Change-Id: I4fc9d716f674df882156cdd1763b8d43204e7d90

commit 24714876f76207a5fc7d72e88bd1f9c57909fcdb
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 19:06:31 2014 -0700

    NCBC-606: Catch and log ServerUnavailableException during failover

    Change-Id: Ie0f74d2bf69cd43ea399762fc39e7ca7f305c11c
    Reviewed-on: http://review.couchbase.org/41384
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit a2cd618f125fc330db742bcb6879c85741e1b3e0
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:43:48 2014 -0700

    NCBC-638: If Server is null when sending an operation then retry operation

    Change-Id: Idfee0ab2852c51cb9c7de1912374d64ee1633be7
    Reviewed-on: http://review.couchbase.org/41374
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 2ca77982d595f15cfa7fe5aa0b97902c94b5f120
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:41:13 2014 -0700

    Update unit tests to use camel cased property values when applicable

    Change-Id: I80ae31e2bf58809574dba965063b21ffe48dc5ec
    Reviewed-on: http://review.couchbase.org/41373
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 33fc515c0248fbeb56fa7123cea954872e4c9b11
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Thu Sep 11 16:35:10 2014 -0700

    NCBC-637: When CCCP fails ensure NRE is not thrown

    Change-Id: I5ef0dbef8298675ceb92e9eb91c49303269133a2
    Reviewed-on: http://review.couchbase.org/41371
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 1e9fff367fa3eefd19069fdfa1dfaae14b4b2a42
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 23:36:43 2014 -0700

    NCBC-586: Make AddRef and Release on IBucket not public

    This commit adds an internal interface IRefCountable with AddRef and
    Release members, so that the two methods are not publically callable on an
    IBucket implementation.

    Change-Id: I11f628094aa1f65ff4780f42e00bbfb088e78898
    Reviewed-on: http://review.couchbase.org/41354
    Reviewed-by: Brett Lawson <brett19@gmail.com>
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 983b94b8acbce8eee11314542d8dcdfbb3c03e0d
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 22:34:37 2014 -0700

    NCBC-636: Make outgoing ContractResolver default to camel casing

    This commit makes the incoming and outgoing JSON IContractResolver's type
    configurable from the ClientConfiguration object. The default settings are
    to use a CamelCasedNameContractResolver for both serialization and
    deserilization.

    Change-Id: I3a82f95d5f4608aa8d05e4141a1a089bc300d4e7
    Reviewed-on: http://review.couchbase.org/41353
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 77b78aaa1550c2ea9e0dfa4085fad2cd24814721
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 18:06:55 2014 -0700

    NCBC-635: Provide better failure support for N1QL Query API

    Change-Id: I16d9a44ba653f8846b22e7c679681ab262e1e6bc
    Reviewed-on: http://review.couchbase.org/41351
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Jeffry Morris <jeffrymorris@gmail.com>

commit 424a10647f5d6dd21e603fbfe8bb98de305f9491
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:43:24 2014 -0700

    NCBC-634: Ensure that CancellationTokenSource is Disposed

    This fix ensures that the CTS will be explicitly disposed so that the
    object does not live through to finalization.

    Change-Id: I39545b97a3c934f2c0a93587fa7147b32d2bd043
    Reviewed-on: http://review.couchbase.org/41335
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8bd607e8cb545ace5102edf1accb5f862b5caf74
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 15:12:17 2014 -0700

    NCBC-632: Make JsonSerializerSettings configurable

    Change-Id: I21c5620a2192217faa68a62cf440a2a8f3dc8893
    Reviewed-on: http://review.couchbase.org/41331
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit bebae67219460e2e3a69ad2d56c9f98afc824190
Author: jeffrymorris <jeffrymorris@gmail.com>
Date: Wed Sep 10 14:26:25 2014 -0700

    NCBC-626: HttpRequestException in queries after rebalance out

    Change-Id: I0eed00d956f231c3826a54da6a983586618d126d
    Reviewed-on: http://review.couchbase.org/41327
    Tested-by: Jeffry Morris <jeffrymorris@gmail.com>
    Reviewed-by: Brett Lawson <brett19@gmail.com>

commit 8db6085e9aecb23cd6defb672f7ec0548536d9de
Author: jeffrymorris <jeffrymorris@gmail.com>

http://www.couchbase.com/issues/secure/attachment/21870/NCBC-649_gerrit.7z


please see screenshots




[NCBC-648] Forcing JsonSerialization on non-nullable ValueTypes fails Created: 18/Sep/14  Updated: 18/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If you try do to a Set on a non-nullable ValueType, NewtonSoft will throw an exception with the following error message returned in the IResult object: "No JSON content found and type 'System.Int32' is not nullable. Path '', line 0, position 0."

The work around is to use a nullable valuetype instead, for example use bucket.Insert<int?>(..) instead of bucket.Insert<int>(..).




[NCBC-647] Markdown Links not Rendered Created: 18/Sep/14  Updated: 18/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: docs
Affects Version/s: 1.3.9
Fix Version/s: 1.3.10

Type: Bug Priority: Minor
Reporter: Chris Malarky Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There are some links not being rendered correctly because of missing text between the brackets in the markdown.

Three in configuring-logging.markdown:

[](https://github.com/couchbase/couchbase-net-client). If you use these Visual
[](http://logging.apache.org/log4net/release/manual/configuration.html).
See [](http://nlog-project.org/wiki/Configuration_file) for more NLog

And one in packages-for-the.net-client-library.markdown (though I can't see this section in the rendered docs):
[](https://www.couchbase.com/develop/net/current/). The download package






[NCBC-646] Couchbase SDK 1.3.X and 2.X are incompatible from a data (namely flags/transcoder) perspective. Created: 17/Sep/14  Updated: 17/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
While running some unit tests on 1.3.X that used keys inserted by 2.X I noticed that the 1.3.X client could not read values written by 2.X. The problem lies with transcoding and specifically that the 2.X client doesn't respect the type of encoding set in the flags by 1.3.X.




[NCBC-645] [ssl] test Rb2Out-HYBRID stuck with 'Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException' in client logs Created: 17/Sep/14  Updated: 17/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive NCBC-645.zip     Text File test_logs.txt    

 Description   
c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
ig.ini -I sdkd.args --variants HYBRID -d all:debug
[3.21 INFO] (BRun run:436) Initializing S3
[7.80 INFO] (BRun run:441) Initializing history database

============================================================
Running Rb2Out-HYBRID: 2014-09-17 02:29:58
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\Rb2Out-HYBRID\09-17-14\0341
98\28852a8b5890fe94895277f14f7f4f66

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
 \ --rebalance-mode out --workload HybridWorkloadGroup
 \ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
 \ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
 \ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase
        --cluster_useSSL=True

[447.89 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[448.91 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 100%
[449.93 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[449.94 INFO] (Scenario run:97) CHANGE phase done
[449.95 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[539.95 INFO] (Scenario run:105) REBOUND done. Will collect results
[539.95 DEBUG] (Handle sendMessageAsync:183) > CANCEL@103.72
[539.97 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 103
[539.97 DEBUG] (Handle receiveMessage:158) < CANCEL@103.72
[539.97 DEBUG] (Handle sendMessageAsync:183) > CANCEL@104.72
[539.98 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 104
[2705.74 DEBUG] (Handle receiveMessage:158) < CANCEL@104.72
[2706.66 DEBUG] (Handle sendMessageAsync:183) > CANCEL@105.72
[2707.19 INFO] (SDKD log:137) [Sdkd.Control|Info] Cancelling command 72 for Handle 105
Terminate batch job (Y/N)? Y




part of the client logs when test stuck(02:37 - 03:14)

2014-09-17 02:37:18,228 [7] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 7
2014-09-17 02:37:18,227 [47] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 40d8fbd1-7b5d-4334-a688-065d2575cca6 on 10.3.3.206:11207 - a17e5df5-7c63-4030-ba02-41309e68ea55
2014-09-17 02:37:18,225 [21] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: abf3e988-cbfc-4795-8523-325094b9e719 on 10.3.121.134:11207 - 624b4df1-35dc-4842-8a23-7db2310b001b
2014-09-17 02:37:18,232 [50] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#808 on thread 50
2014-09-17 02:37:18,230 [9] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 9
2014-09-17 02:37:18,231 [6] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#820 on thread 6
2014-09-17 02:37:18,233 [43] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#828 on thread 43
2014-09-17 02:37:18,253 [18] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#840 on thread 18
2014-09-17 02:37:18,252 [28] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#800 on thread 28
2014-09-17 02:37:18,254 [53] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#798 on thread 53
2014-09-17 02:37:18,258 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#870 on thread 42
2014-09-17 02:37:18,258 [46] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 981 for key SimpleKeyREP5537REP5537REP5537REP5537REP5537 - rev798
2014-09-17 02:37:18,210 [49] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 515 for key SimpleKeyREP5382REP5382REP5382REP5382REP5382 - rev870
2014-09-17 02:37:19,297 [59] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [13] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [62] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [20] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,299 [12] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,297 [15] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.135:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,301 [61] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,298 [60] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 02:37:19,300 [11] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:51,391 [7] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 427 for key SimpleKeyREP5352REP5352REP5352REP5352REP5352 - rev870
2014-09-17 03:14:51,391 [30] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 226 for key SimpleKeyREP5697REP5697REP5697REP5697REP5697 - rev870
2014-09-17 03:14:51,391 [24] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 958 for key SimpleKeyREP5964REP5964REP5964REP5964REP5964 - rev817
2014-09-17 03:14:51,391 [48] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 456 for key SimpleKeyREP5799REP5799REP5799REP5799REP5799 - rev810
2014-09-17 03:14:51,391 [8] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 461 for key SimpleKeyREP6028REP6028REP6028REP6028REP6028 - rev799
2014-09-17 03:14:51,334 [49] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5382REP5382REP5382REP5382REP5382 using server 10.3.121.134:11207
2014-09-17 03:14:51,391 [44] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 246 for key SimpleKeyREP5854REP5854REP5854REP5854REP5854 - rev870
2014-09-17 03:14:51,391 [6] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 34 for key SimpleKeyREP5909REP5909REP5909REP5909REP5909 - rev820
2014-09-17 03:14:51,391 [9] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 728 for key SimpleKeyREP5752REP5752REP5752REP5752REP5752 - rev800
2014-09-17 03:14:51,391 [43] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 554 for key SimpleKeyREP5750REP5750REP5750REP5750REP5750 - rev828
2014-09-17 03:14:51,391 [50] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 109 for key SimpleKeyREP5503REP5503REP5503REP5503REP5503 - rev808
2014-09-17 03:14:51,391 [18] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 633 for key SimpleKeyREP5670REP5670REP5670REP5670REP5670 - rev840
2014-09-17 03:14:51,391 [28] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 825 for key SimpleKeyREP5706REP5706REP5706REP5706REP5706 - rev800
2014-09-17 03:14:51,391 [53] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 964 for key SimpleKeyREP5289REP5289REP5289REP5289REP5289 - rev798
2014-09-17 03:14:51,391 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 610 for key SimpleKeyREP5850REP5850REP5850REP5850REP5850 - rev870
2014-09-17 03:14:51,334 [46] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5537REP5537REP5537REP5537REP5537 using server 10.3.3.206:11207
2014-09-17 03:14:51,631 [45] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5291REP5291REP5291REP5291REP5291
2014-09-17 03:14:51,631 [47] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5660REP5660REP5660REP5660REP5660
2014-09-17 03:14:51,631 [21] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5520REP5520REP5520REP5520REP5520
2014-09-17 03:14:51,632 [52] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation succeeded 0 for key SimpleKeyREP5089REP5089REP5089REP5089REP5089
2014-09-17 02:37:19,300 [63] ERROR Couchbase.Views.ViewClient - System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.3.121.136:18092
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
2014-09-17 03:14:54,754 [7] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5352REP5352REP5352REP5352REP5352 using server 10.3.3.206:11207
2014-09-17 03:14:54,756 [49] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: c1a7c0f9-9ae6-406a-870f-221d1cd5abc7 | 10.3.121.134:11207 | [0, 2] - 45e4bde8-afd4-4084-b4fe-1952631c5f47 - Disposed: False
2014-09-17 03:14:54,760 [18] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5670REP5670REP5670REP5670REP5670 using server 10.3.3.206:11207
2014-09-17 03:14:54,760 [28] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5706REP5706REP5706REP5706REP5706 using server 10.3.3.206:11207
2014-09-17 03:14:54,762 [46] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: dfdafeeb-7f31-4104-94ea-a7c9c951aa6e | 10.3.3.206:11207 | [0, 2] - 728e6fe1-8778-4acd-b15d-832d832b0a1a - Disposed: False
2014-09-17 03:14:54,760 [53] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5289REP5289REP5289REP5289REP5289 using server 10.3.3.206:11207
2014-09-17 03:14:54,761 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5850REP5850REP5850REP5850REP5850 using server 10.3.3.206:11207
2014-09-17 03:14:54,749 [20] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,750 [15] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [59] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,747 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,748 [62] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,760 [50] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5503REP5503REP5503REP5503REP5503 using server 10.3.121.134:11207
2014-09-17 03:14:54,750 [12] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,758 [9] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5752REP5752REP5752REP5752REP5752 using server 10.3.3.206:11207
2014-09-17 03:14:54,754 [30] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5697REP5697REP5697REP5697REP5697 using server 10.3.121.134:11207
2014-09-17 03:14:54,752 [60] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,753 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - trying again: 1
2014-09-17 03:14:54,755 [48] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP5799REP5799REP5799REP5799REP5799 using server 10.3.3.206:11207



 Comments   
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you define "stuck"? Seeing exceptions in the logs is expected, but if the client has stopped completely, this is definitely a blocking issue.

-Jeff
Comment by Andrei Baranouski [ 17/Sep/14 ]
yes, the client has stopped completely
 I only got a new batch of activities (in logs), after I click Ctrl+c in terminal where I run the tests.
That's all I can tell you now. let me know if you need some detail on this issue
Comment by Jeff Morris [ 17/Sep/14 ]
Hi Andrei -

Can you do a "Debug->Break All" and look at the threads to see where it's blocking? I am guessing it's on one of the Wait()'s that doesn't have a timeout set.

Thanks,

Jeff




[NCBC-644] ops/sec is not restored to the expected value even when warmup completed (restart 3/4 nodes) Created: 17/Sep/14  Updated: 17/Sep/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File c8a06f8889f4ee9408205f024dabd0fe-CB.html     HTML File c8a06f8889f4ee9408205f024dabd0fe-MC.html     Microsoft Excel report (38).xls    

 Description   
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/97/consoleFull

Running SvcRestart-HYBRID: 2014-09-17 01:47:02
    service/restore_delay=5
    workload=HybridWorkloadGroup
    service/name=SYSV_SERVICE
    service/action=kill
    service/count=3
    testcase=ServiceFailureScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-17-14\031622\c8a06f8889f4ee9408205f024dabd0fe

    To re-run the test, copy/paste the following into the shell.
    You may also copy/paste (except the first line) into an argfile
    8<----------------------------------------
    ./stester \
        --service-restore_delay 5 --workload HybridWorkloadGroup
 \
        --service-name SYSV_SERVICE --service-action kill --service-count
 \
        3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
 \
        C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
 \
        --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
 \
        --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
 \
        --cluster_ssh-username=root --cluster_ssh-password=couchbase
    ---------------------------------------->8
[1587.82 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution
[1587.91 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: Execute '/etc/init.d/couchbase-server stop' on 3 nodes waiting 1 seconds between each invocation sleep for 5 seconds and restore the nodes. Restoring is peformed by: re-active the services by invoking /etc/init.d/couchbase-server starton each of the nodes. Rebound for 90 seconds.


..............


[1687.17 INFO] (SDKD log:137)  "ViewName": "test_view"
[1687.17 INFO] (SDKD log:137) } 
[1717.17 INFO] (Scenario run:86) Starting CHANGE phase
[1717.17 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.135
[1721.05 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409 
[1722.70 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@6dc8f3cd
[1723.71 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.136
[1726.96 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@4271c5bc
[1727.98 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.3.206
[1731.62 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1bb1deea
[1732.63 INFO] (Scenario sleepAndDo:109) Sleeping 5 seconds
[1737.63 INFO] (Scenario executeChange:134) Turning on services for nodes
[1737.63 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.135
[1739.48 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@ab245dc
[1740.53 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.136
[1742.20 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@5c538b31
[1743.25 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.3.206
[1744.92 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@66de04cd
[1745.93 INFO] (Scenario run:97) CHANGE phase done
[1745.93 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[1751.36 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of an object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 268
[1835.93 INFO] (Scenario run:105) REBOUND done. Will collect results
[1835.93 DEBUG] (Handle sendMessageAsync:183) > CANCEL@278.597



please see report c8a06f8889f4ee9408205f024dabd0fe-MC.html

warmup should not take more than a minute(it should be even less 20 seconds, only 1600 items are in the bucket) but within 90 seconds ops/sec statistic is extremely slow.

in client logs I see a large number of exceptions:
2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 | 10.3.121.135:11210 | [0, 2] - 8f050465-bd1e-4798-9e75-b7dced55a859 - Disposed: False
2014-09-17 01:49:19,230 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3692REP3692REP3692REP3692REP3692 using 313 on 10.3.121.135:11210
2014-09-17 01:49:19,230 [27] WARN Couchbase.IO.ConnectionBase - Opcode=Set | Key=SimpleKeyREP3692REP3692REP3692REP3692REP3692 | Host=10.3.121.135:11210
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
2014-09-17 01:49:19,230 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: be174e18-0c3f-4c4a-ab65-5ac8575468a8 on 10.3.121.135:11210 - 8f050465-bd1e-4798-9e75-b7dced55a859
2014-09-17 01:49:19,231 [27] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.134:11210
2014-09-17 01:49:19,231 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 4be40fbd-8d82-49de-931f-598e6276e7cd | 10.3.121.134:11210 | [0, 2] - b7812f0a-380b-41a4-92f4-ac1db0e0f3a0 - Disposed: False
2014-09-17 01:49:19,231 [27] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.134:11210
2014-09-17 01:49:19,228 [31] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 457 for key SimpleKeyREP3796REP3796REP3796REP3796REP3796 - rev4
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP3796REP3796REP3796REP3796REP3796 using server 10.3.121.135:11210
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: e63556d1-caf1-44fc-8f81-11a670a45a9b | 10.3.121.135:11210 | [0, 2] - 3cd6215b-e17c-493c-bf94-885b7cafaf1a - Disposed: False
2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP3796REP3796REP3796REP3796REP3796 using 457 on 10.3.121.135:11210
2014-09-17 01:49:19,231 [31] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP3796REP3796REP3796REP3796REP3796 | Host=10.3.121.135:11210
System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at Couchbase.IO.Strategies.EapConnection.Send[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 38
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: e63556d1-caf1-44fc-8f81-11a670a45a9b on 10.3.121.135:11210 - 3cd6215b-e17c-493c-bf94-885b7cafaf1a
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.Core.Server - Sending using server 10.3.121.136:11210
2014-09-17 01:49:19,231 [31] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 092717c9-ef47-41df-8a81-839302b58bde | 10.3.121.136:11210 | [0, 2] - df9f7db5-1e94-4d4e-ac0d-441ff1b9c698 - Disposed: False
2014-09-17 01:49:19,231 [31] INFO Couchbase.IO.ConnectionBase - Sending key using 0 on 10.3.121.136:11210



 Comments   
Comment by Andrei Baranouski [ 17/Sep/14 ]
https://s3.amazonaws.com/bugdb/jira/NCBC-644/78bc58e9/NCBC-644.zip




Generated at Tue Sep 23 23:02:16 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.