[NCBC-655] .NET Client seeing many pauses during Rebalance Created: 23/Sep/14  Updated: 21/Oct/14  Resolved: 20/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: 1.3.9
Fix Version/s: backlog-1.0

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Jeff Dillon
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
I have been benchmarking the Java client vs .NET client. During a rebalance, I see frequent pauses within the .NET client that greatly impacts overall throughput. These pauses are not seen in the Java client with equivalent code (gets). I've captured a 30 second video as well as a 30 second network capture, attached. It is connecting to a 3 node 2.5.1 cluster where I add or remove a node, and rebalance.

ASK: Why is the .NET client pausing for 10-15 seconds each minute during a rebalance?

http://screencast.com/t/kNyXkauaEPK3

Here is the relevant C# code, performing a GetWithCas with random keys in a loop:

private static void Main(string[] args)
        {
         
            var config = new CouchbaseClientConfiguration();

            config.Bucket = "TestBucket";
            config.Username = "";
            config.Password = "";

            config.SocketPool.MaxPoolSize = 20;
            config.SocketPool.MinPoolSize = 20;
            config.RetryCount = 10;
            config.HeartbeatMonitor.Interval = 30000;
            config.SocketPool.ReceiveTimeout = TimeSpan.FromMilliseconds(2500);
            config.RetryTimeout = TimeSpan.FromMilliseconds(1000);

            config.Urls.Add(new Uri("http://10.4.2.57:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.107:8091/pools"));
            config.Urls.Add(new Uri("http://10.4.2.108:8091/pools"));

            var Client = new CouchbaseClient(config);

            // create threads to read random documents:

            ParallelOptions parallelOptions = new ParallelOptions();
            parallelOptions.MaxDegreeOfParallelism = 1;
            Parallel.For((long)0, 1, parallelOptions, i =>
            {
                for (int j = 100071; j < 1000000; j++)
                {
                    string key = "Couch_" + j.ToString();
                    var casVal = Client.GetWithCas(key);
                    if (casVal.Result != null)
                    {
                        Console.WriteLine("j=" + j.ToString());
                    }
                }
            });

            Client.Dispose();
        }
    }


FYI, here is the corresponding Java code that does not paused during a rebalance:

public static void main(String[] args) throws Exception {
        new App().run();
    }
    
    private void run() throws InterruptedException, ExecutionException {
        int numThreads = 1;
        
        ExecutorService pool = Executors.newFixedThreadPool(numThreads);
        List<Future> fs = new ArrayList<Future>(numThreads);
        for (int i=0; i<numThreads; i++) {
            fs.add(pool.submit(new Reader(i+1, 5000000)));
        }
        for (Future f : fs) {
            f.get();
        }
    }

    private class Reader implements Runnable {
        
        private int id;
        private long runningTime;
        
        public Reader(int id, long runningTime) {
            this.id = id;
            this.runningTime = runningTime;
        }
        
        @Override
        public void run() {
            long n=100071;
            long start, end=0;
            long initTime = System.currentTimeMillis();
            List<URI> uris = null;
            try {
                uris = Arrays.asList(new URI("http://10.4.2.57:8091/pools"));
            } catch (URISyntaxException e) {
                
                e.printStackTrace();
            }
            String bucketName = "TestBucket";
            String result = "";
            String password = "";
            CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
            builder.setReadBufferSize(20000);
            builder.setOpTimeout(2500);
            builder.setFailureMode(FailureMode.Redistribute);
            builder. setOpQueueMaxBlockTime(1000);
            CouchbaseConnectionFactory connectionFactory = null;
            CouchbaseClient client = null;
            try {
                connectionFactory =
                builder.buildCouchbaseConnection(uris, bucketName, "", "");
                client = new com.couchbase.client.CouchbaseClient(connectionFactory);
            } catch (IOException e) {
                e.printStackTrace();
            }
            
            while (true) {
                String key = "Couch_" + n;
                start = System.currentTimeMillis();
                
                if (client.gets(key) !=null) {
                    System.out.println("N=" + n);
                }
                
                n++;
                if (runningTime>0 && (end-initTime)>runningTime) {
                    break;
                }
            }
        }
    }


 Comments   
Comment by Jeff Morris [ 23/Sep/14 ]
Jeff -

Thanks for doing this :) Not sure why the client pauses - I have some ideas, though; I'll have to take a deeper look.

-Jeff
Comment by Jeff Morris [ 20/Oct/14 ]
Pull request in Gerrit: http://review.couchbase.org/#/c/42242/

Will merge after CR.
Comment by Jeff Morris [ 21/Oct/14 ]
Jeff D -

Would you mind verifying this fix against your benchmark?

git fetch ssh://jmorris@review.couchbase.org:29418/couchbase-net-client refs/changes/42/42242/3 && git checkout FETCH_HEAD

-Jeff M




[NCBC-685] tests stuck when reb out with spamming 'No connections currently available on...' in the logs Created: 14/Oct/14  Updated: 19/Oct/14  Resolved: 15/Oct/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-685.png     Zip Archive Rb2In-HYBRID.zip     Zip Archive Rb2OutEpt-HYBRID.zip     Zip Archive Rb2Out-HYBRID.zip    

 Description   
run with https://github.com/couchbase/couchbase-net-client/commit/1bb5cc0c2f64a93d28cbc5fe057d83b24d98c0bb commit

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/148/console

Running Rb2OutEpt-HYBRID: 2014-10-14 08:44:58
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/ept=true
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1209\Rb2OutEpt-HYBRID\10-14-14\056698\448ba4c89ea8854d954e1f2b048073d0

seems like the problem is with view queries that are requested on rebalanced out nodes.



2014-10-14 08:51:34,608 [59] ERROR Couchbase.Views.ViewClient - System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Couchbase.Views.ViewClient.Execute[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Views\ViewClient.cs:line 91
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 8
2014-10-14 08:51:34,766 [8] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - e0956315-f52e-4159-bcff-655aee01b6a9
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - b5b71f8b-b6d2-4d4e-bf47-cde9f7d595c6
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e8fd62a2-ac1c-43a5-acf1-0cb158d46a9c
2014-10-14 08:51:34,768 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - a153baaf-e71a-4fad-acb1-b36535c0f681
2014-10-14 08:51:34,768 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - a24c2990-62e7-4304-865f-94211fc1747d
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - c6c9c860-5364-47e1-9c56-e9737b8983a1
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - 3775fcd4-bb20-4d50-bebc-2afc9bccf9c1
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 5edbbede-df88-4d29-8cb8-4e3fd31a642d
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - b19a1773-2c38-4f1e-b5c6-5c1782bdf4af
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 19f5edee-12bc-4d43-933d-7c476e034aed
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - f892df91-3644-446d-b8e9-30e18e5053b5
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 9dd5bbfc-e400-4d40-a493-ebcc3a112e95
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - cb74cdd7-ef18-411b-a054-65a7239d4f19
2014-10-14 08:51:34,770 [8] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 8
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 11
2014-10-14 08:51:34,773 [11] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - b5d4bcf1-12ac-4981-8a57-3a4345a1ec0b
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - ad92dfaf-7035-4561-9788-6047552256c6
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 6a748388-aa39-4cb2-9788-495bbbcba48d
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - b59daeeb-602b-43b9-9a5a-35967db9a434
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - 7d689af7-aef3-46a6-aebe-7f51ecfa2073
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 873a7539-d603-46c1-a132-7f113ac83e85
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - e24c15c1-4692-4d15-a064-1d4850194518
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - aabec85d-8df3-4caa-9205-9a7477710bca
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - 07476198-0fb5-4af9-8eef-879c440c249b
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 8540599a-b39a-4f6e-b948-2fb3ddc1c24a
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - d17ebe9b-a444-4eec-96fb-583ab7b34f38
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 19567acf-71a3-43c4-99f3-2ee5145aa265
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - 5fc38936-49ab-4ca5-a8b4-a578f044f581
2014-10-14 08:51:34,774 [11] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,774 [8] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 11
2014-10-14 08:51:34,774 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 9
2014-10-14 08:51:34,775 [9] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - 076b028c-1934-449a-ad17-1fc76ca3e5fb
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - 7a26c404-76a4-4435-9cd9-d322df61efb4
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e3bb78a5-8623-495e-98db-40e43c40d634
2014-10-14 08:51:34,774 [8] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e38251e2-0916-48bf-bb9a-761fc1e0ac5e
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - e822c4a9-a307-4375-9d43-2fc06618867a
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 545ea7df-77a4-44a0-8c9d-1a5755062255
2014-10-14 08:51:34,777 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - c2aff532-9e7e-42c9-9608-a8adfcb86dd0
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - de2c6395-5e6b-48db-b81e-e5c41521abb1
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - b84ef574-e7a3-4f16-a405-72cab7e6999c
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 7be62279-00fa-442e-8137-651ae402dd75
2014-10-14 08:51:34,780 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,780 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 26b996bc-6685-4716-8237-cd3799b8a792
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 2834d4f4-0280-4c8d-ba62-2d491ca483c7
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - 70acb54e-5d71-40f1-98d8-039b5e05c1d2
2014-10-14 08:51:34,781 [9] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 9
2014-10-14 08:51:34,781 [11] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,781 [11] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,781 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 14
2014-10-14 08:51:34,784 [14] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - b3673bf7-51f1-42a5-9395-5ef22b7c7fcf
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - e2da8b66-96f9-4ffb-8ce7-84521cfa8861
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 0ebcd0c1-52e2-4592-b026-9d19bce12b94
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - a724690b-6ab1-4049-977c-80a1276e4239
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - 87518c99-7bdd-4e9b-8e6c-363fbe5c12d4
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 58dbd214-f9ac-45e8-b0f1-9170a943a7cf
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - a87e0542-4993-4458-bb2c-132473835204
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 97f6e739-7ac0-41c9-8e62-089efd454bd3
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - c401faf6-0866-44e8-a9b2-0612b933808a
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - eb8e57a2-f73f-403e-a514-fe73710ca7c7
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 96f5c737-aad1-4774-9a2f-590678ac6e62
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 18c5670e-b27f-474c-8100-53962e3c7009
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - cb8327b9-b45c-451b-b936-393ebd41b1f8
2014-10-14 08:51:34,785 [14] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 14
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 13
2014-10-14 08:51:34,798 [13] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - a29394e2-5b67-410a-8fa6-68a263a2351a
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - 554f7d15-4c08-485b-8235-6588e2cea61e
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 6477b8a9-f68c-47da-b865-8fae30bdd3e9
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - d122c532-69c7-49a1-b867-ea5bc0f50cd5
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - a5212957-d7b4-4e8a-8de2-c134e8b347cd
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - d54a3668-c9da-4e32-82ea-e451cfd3355e
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - 07dd9661-1f98-4964-9f47-a6882ae9968f
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 54f98a31-fb83-4722-b7db-2d8c2620f131
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - d15a2d13-ae10-4280-a963-db384cfd6458
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 94954350-41c8-4001-932b-02b683ca5c60
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 9d7480c8-a451-4b73-9cb2-75ae2dc0dacb
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 251020be-b0e7-489e-888d-05ff93bbf0ea
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - d0a8de8e-89f8-46f5-b08c-1bd5f4982896
2014-10-14 08:51:34,799 [13] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 13
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:37,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:39,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:42,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:44,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:47,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:49,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:52,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:54,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:57,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:59,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:02,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:04,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:07,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:09,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:12,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:14,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:17,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:19,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:22,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:24,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:27,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:29,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:32,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:37,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:39,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:42,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:44,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:47,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:49,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:52,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:54,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:57,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:59,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:02,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:04,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:07,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:09,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:12,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:14,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:17,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:19,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:22,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:24,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:27,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
....

 Comments   
Comment by Andrei Baranouski [ 14/Oct/14 ]
https://s3.amazonaws.com/bugdb/jira/NCBC-685/e82fc475/NCBC-685.zip
Comment by Jeff Morris [ 14/Oct/14 ]
It looks like the test harness is disposing the cluster?!:

2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 13
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False

Not sure why the ConnectionPool logger indicates the it's not disposed, though.
Comment by Jeff Morris [ 14/Oct/14 ]
>>seems like the problem is with view queries that are requested on rebalanced out nodes.

The connection pool is only used for memcached ops; views just use the server endpoints to construct the url in a round-robin fashion.
Comment by Andrei Baranouski [ 15/Oct/14 ]
Hi Jeff,

I'm able to reproduce it in command line on 172.23.107.172

stack trace there

  [External Code]
> Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 123 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Couchbase.IO.IConnectionPool.Acquire() Line 178 C#
  Couchbase.dll!Couchbase.IO.Strategies.DefaultIOStrategy.Execute<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 40 C#
  Couchbase.dll!Couchbase.Core.Server.Send<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 90 C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 95 C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.Get<string>(string key) Line 967 C#
  Sdkd.dll!Sdkd.Commands.GetCommand.ExecuteSingle(string key, string value) Line 17 C#
  Sdkd.dll!Sdkd.Commands.KVCommand.Run() Line 24 C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.Buckets.CouchbaseBucket cli) Line 266 C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 146 C#
  [External Code]
Comment by Jeff Morris [ 15/Oct/14 ]
Thanks Andrei. I know where this is happening, I just don't know most importantly WHY it's occurring. Basically, the connection pool is empty and all threads are waiting for a connection to become available. Since the PoolConfiguration.MaxSize has been reached, no more connections are being created. Note that there is a timeout for the max amount of time that a connection can be used, so either this isn't be honored (doubtful) or more like the connections are be destroyed somehow and not reused.

-Jeff
Comment by Jeff Morris [ 15/Oct/14 ]
This would cause a connection to not be re-added to the pool (since the auth failed on it while it was being initialized):

Line 1495903: 2014-10-15 14:47:44,782 [16] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: db69c7dd-2d91-4013-a086-5d739867e02b | 10.3.4.144:11210 | [0, 1] - 3dd67e96-c247-4870-aab1-d7d867426c4b - Disposed: False
Line 1495904: 2014-10-15 14:47:44,782 [16] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authenticating socket db69c7dd-2d91-4013-a086-5d739867e02b
Line 1495928: 2014-10-15 14:47:47,283 [16] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authentication for socket db69c7dd-2d91-4013-a086-5d739867e02b failed: Operation timed out: the timeout can be configured by changing the PoolConfiguration.OperationTimeout property. The default is 2500ms.
Comment by Jeff Morris [ 15/Oct/14 ]
Patch: http://review.couchbase.org/#/c/42194/
Comment by Andrei Baranouski [ 16/Oct/14 ]
good results for the test and it doesn't stuck)

http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2Out-HYBRID/10-16-14/032835/183bac130ab3cd60442dfdcc92f9b61e-MC.html


but there are some unexpected statistics in other tests. will summarize them soon
Comment by Andrei Baranouski [ 16/Oct/14 ]
for some cases, for exmaple Rb2OutEpt I still see considerably smaller ops/sec view requests values during and after rebalance out
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-16-14/032248/0fc0dcd7fc34e7ad9aa92e6615c36329-CB.html
and MEMD:GENERIC error afte simple rebalance out 4-2 nodes
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-16-14/032248/0fc0dcd7fc34e7ad9aa92e6615c36329-MC.html

latency increased after rebalance Rb2In, it should be less than
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-16-14/034067/f32ec4d1e53a4024e529676904f6d216-MC.html


MEMD:GENERIC error afte failover 1 EPT node and rebalance out
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-16-14/034611/981225e9aa00099891b40ae02ad922bb-MC.html


results are from http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/35/consoleFull

Comment by Jeff Morris [ 17/Oct/14 ]
These are run off of my machine using SDKD (see attachements above). Overall the results looks significantly better: MEMD:GENERIC and EVBUCKET go down after rebound along with latency, while generally performance improves.
Comment by Andrei Baranouski [ 18/Oct/14 ]
Hi Jeff, I think we can close this ticket because the original problem has been fixed
and to keep track of all the inconsistencies / problems that we see in different runs VS envs I created a separate ticket NCBC-690
Comment by Jeff Morris [ 18/Oct/14 ]
Ok, sounds good.




[NCBC-689] Disable ServicePointManager.Expect100Continue by setting to false Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Improvement Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Overrides the default service manager behavior by disabling 100-continue headers being sent by the HttpClient and WebClient; which may incur a small performance hit.

 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
http://review.couchbase.org/#/c/42229/




[NCBC-688] Intermittent deadlock during perf test Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Closed
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Test Blocker
Reporter: Li Yang Assignee: Jeff Morris
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Running .NET iOS perf test on Xamarin studio.

Attachments: File deadlock.tiff    

 Description   
This problem happens from time to time, usually in the heavy load testing that the system would freeze without any error or exception.

Pasin did a screen captured (see attached file) that shows the stack trace. At time of error, the test case running is Test03CreateDocsWithAttachments, iteration #16, creation of 10000 documents, each with 500 Bytes size.

 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
Li -

I think this for the Couchbase Lite/mobile project? The Couchbase .NET client library is the desktop/server library.

-Jeff
Comment by Matt Ingenthron [ 16/Oct/14 ]
The Couchbase Lite .NET issue tracker is here: https://github.com/couchbase/couchbase-lite-net/issues
Comment by Li Yang [ 16/Oct/14 ]
Sorry, opened it at the wrong place. The bug is moved to https://github.com/couchbase/couchbase-lite-net/issues/335




[NCBC-687] hello couchbase app missing instruction to add using Couchbase; Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: Cihan Biyikoglu Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
http://docs.couchbase.com/developer/dotnet-2.0/hello-couchbase.html
after nuget - you still need to add using Couchbase; at the top for CouchbaseCluster() to resolve.


 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
This will be updated in the SDK 2.0 GA release.




[NCBC-675] Fixed servers with FQDN (ex: ubuntu.mybox) not properly resolving Created: 07/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Critical
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If a bucket configuration returned from a Couchbase cluster contains a fully qualified domain name instead of an IP address, CouchbaseConfigContext will not properly resolve the endpoint, resulting in a null KeyMapper. Upon any operation that requires the KeyMapper, the Couchbase .Net client will throw a NullReferenceException.

 Comments   
Comment by Jeff Morris [ 09/Oct/14 ]
https://github.com/couchbase/couchbase-net-client/pull/20
Comment by Jeff Morris [ 16/Oct/14 ]
http://review.couchbase.org/41984




Generated at Tue Oct 21 19:43:59 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.