[NCBC-692] BasicStats.MemUsed is too small (int32) for the value that is stored in it Created: 21/Oct/14  Updated: 21/Oct/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0, 2.0.1

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Stacktrace:

AggregateException: Could not bootstrap - check inner exceptions for details.

InnerException: JSON integer 2896300456 is too large or small for an Int32. Path '[0].basicStats.memUsed', line 1, position 9763.




[NCBC-691] Maximum and Default Locking times are misleading Created: 19/Oct/14  Updated: 19/Oct/14

Status: Open
Project: Couchbase .NET client library
Component/s: docs
Affects Version/s: 1.3.9
Fix Version/s: 1.3.10

Type: Bug Priority: Major
Reporter: Chris Malarky Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-12379 Default and Maximum lock time describ... Open

 Description   
The maximum lock in 2.5.1 and 3.0.0 is 30 secs, the default is 15 seconds. Any attempt to set a lock higher than the maximum results in the default being used. The .NET docs need updating to match this behaviour.

 Comments   
Comment by Chris Malarky [ 19/Oct/14 ]
Not sure if 2.0 is affected? Couldn't find a GetWithLock in the docs.
Comment by Jeff Morris [ 19/Oct/14 ]
GetWithLock hasn't been implemented yet it 2.0.




[NCBC-690] significant differences in the results on different platforms(win8/2008/2012) Created: 18/Oct/14  Updated: 21/Oct/14

Status: Reopened
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Run all tests on 2 different platforms:

win2008 http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/151/consoleFull
win2012 http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/44/consoleFull
win8 http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win8_64_andrei/17/consoleFull

Rb2OutEpt-HYBRID
#1 win 2008 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/039627/915d4e48ac032c96c7f4950b52a9f12b-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/039627/915d4e48ac032c96c7f4950b52a9f12b-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/045117/e57f6854971994954036d3eadcbb9816-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/045117/e57f6854971994954036d3eadcbb9816-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/062193/b25405af18dbe9b92bc69923e438b030-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/062193/b25405af18dbe9b92bc69923e438b030-CB.html

DIFFERENCES: MEMD:GENERIC errors & Latency issues on win 2012 after rebalance



Rb2In-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/040984/b227a05c68fd1d51ec065896d1e9e469-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/040984/b227a05c68fd1d51ec065896d1e9e469-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/047439/888d0e863b54a5d2e011aafd4b846e90-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/047439/888d0e863b54a5d2e011aafd4b846e90-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/063854/f028c9f4e5611dcf898555f4fa8a2732-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/063854/f028c9f4e5611dcf898555f4fa8a2732-CB.html

DIFFERENCES: hight latency remains during/after rebalance on win2012 & win8( ok on win2008)


FoEptRb-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/041455/7b08d1d9841e15681bcd3ad74b313e9e-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/041455/7b08d1d9841e15681bcd3ad74b313e9e-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/048177/65d411ff5f3d6b05f01f45ab2824c6e7-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/048177/65d411ff5f3d6b05f01f45ab2824c6e7-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/064372/a638f92c38ad325f28b2217b31821796-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/064372/a638f92c38ad325f28b2217b31821796-CB.html

DIFFERENCES:MEMD:GENERIC errors only on win2012


SvcRestart-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/041737/be7b44ee6715307c78d7cc98c96d3d1f-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/041737/be7b44ee6715307c78d7cc98c96d3d1f-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/048605/7b5bfaac53a620d6d92918319f1d1645-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/048605/7b5bfaac53a620d6d92918319f1d1645-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/064752/f89c6dd6eb392c288174012f97f919af-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/064752/f89c6dd6eb392c288174012f97f919af-CB.html

DIFFERENCES:MEMD:GENERIC errors only on win2012

Notes:
I used the same 4 vms for CB clusters(they are from 10.* LN)
clients:
win2008 from 10.*N
win8 & win2012 are from 172.* LN


Jeff, please let me know how I can help find the cause of these differences



 Comments   
Comment by Jeff Morris [ 18/Oct/14 ]
Andrei -

Can you post the client logs for the ones that show mem:generic errors after rebound?

How can performance go up with high latency? Aren't the two inversely related?: http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/063854/f028c9f4e5611dcf898555f4fa8a2732-MC.html

***Next steps: I think we should do two things***:
a) Re-run the tests above and compare the graphs - if the results are the same, it's likely a platform issue. If the results are different, it may be an issue that only randomly appears.
b) Delete and completely rebuild your cluster VM's - then re-run the tests and compare again.

I know it's a lot of work, but we need to get to the bottom of this...I now SDKD running (on Win8.1) I have yet to get any memd:generic errors after rebound. Without something to pivot on, I don't know what to change.

Lastly, If we don't get something to work with by tomorrow, I'll go through and add instrumentation to all the operations and see if we can detect where the latency issues are coming from.

Thanks for the hard work!

-Jeff
Comment by Andrei Baranouski [ 19/Oct/14 ]
iteration#2 with the same build/client version/vms

Rb2OutEpt-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/018438/d05cbc1d641d6c2223e19a53a6d87bce-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/018438/d05cbc1d641d6c2223e19a53a6d87bce-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/023642/270466b867b66e280e58ff68f1663019-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-18-14/023642/270466b867b66e280e58ff68f1663019-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-19-14/036787/6f2065c5f65ce166b1635233dd34b489-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-19-14/036787/6f2065c5f65ce166b1635233dd34b489-CB.html

the issue "DIFFERENCES: MEMD:GENERIC errors & Latency issues on win 2012 after rebalance" was not reproduced but there is a hight latency for win 8 & win2012 after rebalance


Rb2In-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/019794/e7f9a348baefe9732524816e86e88171-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-18-14/019794/e7f9a348baefe9732524816e86e88171-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-19-14/025533/60e7f1161d7bb33c36c93daa42814f83-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-19-14/025533/60e7f1161d7bb33c36c93daa42814f83-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-19-14/038677/af16ad8c2b3e3630954311f1da34ff01-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-19-14/038677/af16ad8c2b3e3630954311f1da34ff01-CB.html


"DIFFERENCES: hight latency remains during/after rebalance on win2012 & win8( ok on win2008) " reproduced + MEMD:GENERIC on win2012 & win8


FoEptRb-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/020235/8cd558018fa14bd08d5fca6313ab341f-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-18-14/020235/8cd558018fa14bd08d5fca6313ab341f-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-19-14/026173/bb69f8998a9c59f86a8bf20a367666c1-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-19-14/026173/bb69f8998a9c59f86a8bf20a367666c1-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-19-14/039183/16906b487388a5236926fa182e9da017-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-19-14/039183/16906b487388a5236926fa182e9da017-CB.html

"DIFFERENCES:MEMD:GENERIC errors only on win2012" + the same on win8

SvcRestart-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/020528/576f895f54327a9451c0045caa9a9116-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-18-14/020528/576f895f54327a9451c0045caa9a9116-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-19-14/026587/5c4322795342272796b616a17af713a7-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-19-14/026587/5c4322795342272796b616a17af713a7-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-19-14/039559/61814c19baa8cb8ac760f1df34af1dd8-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/SvcRestart-HYBRID/10-19-14/039559/61814c19baa8cb8ac760f1df34af1dd8-CB.html


Not reproduced "DIFFERENCES:MEMD:GENERIC errors only on win2012" but there is issue with hight latency on win8



Another discrepancy in new test for this run

FoEptEject-HYBRID
#1 win 2008
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-18-14/021157/96f3b923f2c650e3ac83658797af1d88-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-18-14/021157/96f3b923f2c650e3ac83658797af1d88-CB.html
#2 win 2012
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-19-14/027413/13f3a2a92cbf002b1114a6269b17d2bf-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-19-14/027413/13f3a2a92cbf002b1114a6269b17d2bf-CB.html
#3 win 8
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-19-14/040390/e60084e33c095d2c5422a0a203cbecc6-MC.html
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptEject-HYBRID/10-19-14/040390/e60084e33c095d2c5422a0a203cbecc6-CB.html

MEMD:GENERIC only in 2012

@Jeff:
"How can performance go up with high latency?"
@Andrei
may be some part of the query is very slow, even though all the rest are just as fast or even faster then before
Comment by Jeff Morris [ 19/Oct/14 ]
>>may be some part of the query is very slow, even though all the rest are just as fast or even faster then before

It could be that the memd:generic are view timeouts (I only give 5secs for a View request), which would account for the higher latency while memcached ops increase.

@Andrei - have you tried rebuilding the VM's?

I think at this point the next steps are to:
1) Add performance logging to the client for Views and Memcached - this will show time between request sent and received.
2) Add more granular logging for memd:generic in SDKD. Mark Nunberg pointed this out and started the implementation during our meeting on Thurs. Remember? Andrei, can you continue this?
3) On the tests showing higher latency & memd:generic, I need the client logs for those specific tests. I can ascertain the type of memd:generic from the log entries.

Once again, _thank you_ very much for your time and effort. I think we are really close here, let's make a final push and get this client released!

-Jeff
Comment by Andrei Baranouski [ 20/Oct/14 ]
will do 1) & 2)

logs for FoEptRb with MEMD:GENERIC error
https://s3.amazonaws.com/bugdb/jira/NCBC-690/da9555cb/NCBC-690_FoEptRb.zip
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-19-14/019202/51ff8414172ae3787fdcfcdd6683a27f-MC.html
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/47/consoleFull
Comment by Jeff Morris [ 20/Oct/14 ]
The memd:generic surprisingly are View request 404's?!:

2014-10-19 22:24:59,171 [9] ERROR Couchbase.Views.ViewClient - System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Couchbase.Views.ViewClient.Execute[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Views\ViewClient.cs:line 91

Here are the Urls being used during the period of time the errors are occurring:

http://10.3.121.136:8092/default/_design/test_design/_view/test_view?
http://10.3.121.135:8092/default/_design/test_design/_view/test_view?
http://10.3.121.134:8092/default/_design/test_design/_view/test_view?
http://10.3.3.206:8092/default/_design/test_design/_view/test_view?

@Andrie - can you ping these and see if they are valid and/or accessible?

I'll add more logging to capture the url's which are specifically failing.

-Jeff
Comment by Jeff Morris [ 20/Oct/14 ]
Andrei -

I added additional logging: http://review.couchbase.org/#/c/42302/

Can you re-run the tests and get me the logs?

Thanks,

Jeff
Comment by Andrei Baranouski [ 21/Oct/14 ]
Hi Jeff,

a) view urls are correct
b) I run FoEptRb test with your commit couple times with slight change

 public void Dispose()
        {
            var totalMilliseconds = m_sw.Elapsed.TotalMilliseconds;
            if (totalMilliseconds > 100)
            {
                Log.Debug(m => m("{0,7:N0} (GCs={1,3}) {2}",
                m_sw.Elapsed.TotalMilliseconds,
                GC.CollectionCount(0) - m_collectionCount, m_text));
            }
            Interlocked.Decrement(ref s_NumOperationTimersStarted);
            m_sw = null;
        }

from time to time I see god results, for example http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-21-14/028883/e1adbe850e5a1693b956ed4c4be67fd4-MC.html
(http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/48/console)

but more often I see picture like http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-21-14/031911/c102a2ed3a5c89e52f82ca4e1661853f-MC.html
(http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/52/console)

2014-10-21 01:54:23,398 [15] DEBUG Wintellect.OperationTimer2 - 2,501 (GCs= 1) Sending SimpleKeyREP4937REP4937REP4937REP4937REP4937 using server 10.3.121.134:11210
2014-10-21 01:54:28,143 [15] DEBUG Wintellect.OperationTimer2 - 2,500 (GCs= 0) Sending SimpleKeyREP5022REP5022REP5022REP5022REP5022 using server 10.3.121.135:11210
2014-10-21 01:56:42,591 [8] DEBUG Wintellect.OperationTimer2 - 3,071 (GCs= 2) Sending SimpleKeyREP1REP1REP1REP1REP1REP1REP1REP1 using server 10.3.121.135:11210
2014-10-21 01:57:01,451 [38] DEBUG Wintellect.OperationTimer2 - 3,066 (GCs= 9) Sending SimpleKeyREP14REP14REP14REP14REP14REP14REP14 using server 10.3.121.136:11210
2014-10-21 01:57:05,815 [42] DEBUG Wintellect.OperationTimer2 - 3,064 (GCs= 9) Sending SimpleKeyREP1REP1REP1REP1REP1REP1REP1REP1 using server 10.3.121.135:11210

2014-10-21 01:59:49,748 [46] ERROR Couchbase.Views.ViewClient - View request error: http://10.3.121.134:8092/default/_design/test_design/_view/test_view? System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Couchbase.Views.ViewClient.Execute[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Views\ViewClient.cs:line 92
2014-10-21 01:59:49,749 [46] DEBUG Wintellect.OperationTimer2 - 4,509 (GCs= 0) Sending View Request: http://10.3.121.134:8092/default/_design/test_design/_view/test_view?

2014-10-21 01:58:16,676 [20] ERROR Couchbase.Views.ViewClient - View request error: http://10.3.3.206:8092/default/_design/test_design/_view/test_view? System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetResponse()
   at Couchbase.Views.ViewClient.Execute[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Views\ViewClient.cs:line 92
2014-10-21 01:58:16,677 [20] DEBUG Wintellect.OperationTimer2 - 5,013 (GCs= 24) Sending View Request: http://10.3.3.206:8092/default/_design/test_design/_view/test_view?




and logs related MEMD:GENERIC

2014-10-21 02:00:51,355 [50] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP7586REP7586REP7586REP7586REP7586 | Host=10.3.3.206:11210
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at Couchbase.IO.Strategies.EapConnection.ReceiveCallback(IAsyncResult asyncResult) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 74
2014-10-21 02:00:51,355 [34] DEBUG Couchbase.CouchbaseBucket - Attempting dispose on thread 34
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.CouchbaseBucket - Current bucket refCount# 1
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.CouchbaseBucket - Removing bucket refCount# 0
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.CouchbaseBucket - Disposing on thread 35
2014-10-21 02:00:51,355 [35] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - d7621b82-d708-46cf-b927-20c8ac9739ff
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 094a36ef-7a54-44b1-98db-dea4d0fd0bb1
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - 93e6cfe9-7bd4-457a-89af-9e27bdb41e55
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - a73ae923-7901-4866-b4a5-5888f7ee31d0
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-21 02:00:51,355 [50] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP7635REP7635REP7635REP7635REP7635 | Host=10.3.121.135:11210
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at Couchbase.IO.Strategies.EapConnection.ReceiveCallback(IAsyncResult asyncResult) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 74
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 7087d96d-25d8-4632-b22c-8a6fe2012cd6
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - fef725a0-c59b-462e-994c-0485825ed819
2014-10-21 02:00:51,355 [35] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 4c82055d-688c-45db-8826-9cb48f0cbc0d
2014-10-21 02:00:51,356 [18] WARN Couchbase.IO.ConnectionBase - Opcode=Get | Key=SimpleKeyREP7637REP7637REP7637REP7637REP7637 | Host=10.3.121.135:11210
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at Couchbase.IO.Strategies.EapConnection.ReceiveCallback(IAsyncResult asyncResult) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\IO\Strategies\EapConnection.cs:line 74

https://s3.amazonaws.com/bugdb/jira/NCBC-690/3db957f3/NCBC-690_timings.zip




[NCBC-689] Disable ServicePointManager.Expect100Continue by setting to false Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Improvement Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Overrides the default service manager behavior by disabling 100-continue headers being sent by the HttpClient and WebClient; which may incur a small performance hit.

 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
http://review.couchbase.org/#/c/42229/




[NCBC-688] Intermittent deadlock during perf test Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Closed
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Test Blocker
Reporter: Li Yang Assignee: Jeff Morris
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Running .NET iOS perf test on Xamarin studio.

Attachments: File deadlock.tiff    

 Description   
This problem happens from time to time, usually in the heavy load testing that the system would freeze without any error or exception.

Pasin did a screen captured (see attached file) that shows the stack trace. At time of error, the test case running is Test03CreateDocsWithAttachments, iteration #16, creation of 10000 documents, each with 500 Bytes size.

 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
Li -

I think this for the Couchbase Lite/mobile project? The Couchbase .NET client library is the desktop/server library.

-Jeff
Comment by Matt Ingenthron [ 16/Oct/14 ]
The Couchbase Lite .NET issue tracker is here: https://github.com/couchbase/couchbase-lite-net/issues
Comment by Li Yang [ 16/Oct/14 ]
Sorry, opened it at the wrong place. The bug is moved to https://github.com/couchbase/couchbase-lite-net/issues/335




[NCBC-687] hello couchbase app missing instruction to add using Couchbase; Created: 16/Oct/14  Updated: 16/Oct/14  Resolved: 16/Oct/14

Status: Resolved
Project: Couchbase .NET client library
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: Cihan Biyikoglu Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
http://docs.couchbase.com/developer/dotnet-2.0/hello-couchbase.html
after nuget - you still need to add using Couchbase; at the top for CouchbaseCluster() to resolve.


 Comments   
Comment by Jeff Morris [ 16/Oct/14 ]
This will be updated in the SDK 2.0 GA release.




[NCBC-686] Use POST when requesting a View if the URI exceeds the maximum character length Created: 15/Oct/14  Updated: 15/Oct/14

Status: Open
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 1.3.9
Fix Version/s: 1.3.10

Type: Bug Priority: Major
Reporter: Jeff Morris Assignee: Jeff Morris
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
From https://forums.couchbase.com/t/getview-need-http-post-instead-of-http-get/1743:

When executing a call extremely similar to the one below I receive the error "Invalid URI: The Uri string is too long".

Code snippet:

CouchbaseClient client;
client.GetView<SomeDataModel>("DesignName", "ViewName", true)
            .Stale(StaleMode.False)
            .Reduce(true)
            .StartKey<object[]>(["someString", 0, "", "", "", 0, ""])
            .EndKey<object[]>(["someString", 9999, "zzzzzzzzzzzzzzzzzzzz", "zzzzzzzzzzzzzzzzzzzz", "zzzzzzzzzzzzzzzzzzzz", 999999999, "zzzzzzzzzzzzzzzzzzzz"]);

If I remove the Start Key and End Key limiters the view will successfully return so I presume that an HTTP GET is being performed and the length of the keys are causing the issue. I used 20 "z"s to represent how long the data could be. For a literal end when there isn't a value specified of course "{" would work, but when values are specified for the strings could be 20 characters long, or possibly even longer.




[NCBC-685] tests stuck when reb out with spamming 'No connections currently available on...' in the logs Created: 14/Oct/14  Updated: 19/Oct/14  Resolved: 15/Oct/14

Status: Closed
Project: Couchbase .NET client library
Component/s: library
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0.0

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Jeff Morris
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File NCBC-685.png     Zip Archive Rb2In-HYBRID.zip     Zip Archive Rb2OutEpt-HYBRID.zip     Zip Archive Rb2Out-HYBRID.zip    

 Description   
run with https://github.com/couchbase/couchbase-net-client/commit/1bb5cc0c2f64a93d28cbc5fe057d83b24d98c0bb commit

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/148/console

Running Rb2OutEpt-HYBRID: 2014-10-14 08:44:58
    rebalance/mode=out
    workload=HybridWorkloadGroup
    rebalance/ept=true
    rebalance/count=2
    testcase=RebalanceScenario
    Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1209\Rb2OutEpt-HYBRID\10-14-14\056698\448ba4c89ea8854d954e1f2b048073d0

seems like the problem is with view queries that are requested on rebalanced out nodes.



2014-10-14 08:51:34,608 [59] ERROR Couchbase.Views.ViewClient - System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Couchbase.Views.ViewClient.Execute[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Views\ViewClient.cs:line 91
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 8
2014-10-14 08:51:34,766 [8] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,766 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - e0956315-f52e-4159-bcff-655aee01b6a9
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - b5b71f8b-b6d2-4d4e-bf47-cde9f7d595c6
2014-10-14 08:51:34,767 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e8fd62a2-ac1c-43a5-acf1-0cb158d46a9c
2014-10-14 08:51:34,768 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - a153baaf-e71a-4fad-acb1-b36535c0f681
2014-10-14 08:51:34,768 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - a24c2990-62e7-4304-865f-94211fc1747d
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - c6c9c860-5364-47e1-9c56-e9737b8983a1
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - 3775fcd4-bb20-4d50-bebc-2afc9bccf9c1
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 5edbbede-df88-4d29-8cb8-4e3fd31a642d
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - b19a1773-2c38-4f1e-b5c6-5c1782bdf4af
2014-10-14 08:51:34,769 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 19f5edee-12bc-4d43-933d-7c476e034aed
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - f892df91-3644-446d-b8e9-30e18e5053b5
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 9dd5bbfc-e400-4d40-a493-ebcc3a112e95
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - cb74cdd7-ef18-411b-a054-65a7239d4f19
2014-10-14 08:51:34,770 [8] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,770 [8] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 8
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 11
2014-10-14 08:51:34,773 [11] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - b5d4bcf1-12ac-4981-8a57-3a4345a1ec0b
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - ad92dfaf-7035-4561-9788-6047552256c6
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 6a748388-aa39-4cb2-9788-495bbbcba48d
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - b59daeeb-602b-43b9-9a5a-35967db9a434
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - 7d689af7-aef3-46a6-aebe-7f51ecfa2073
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 873a7539-d603-46c1-a132-7f113ac83e85
2014-10-14 08:51:34,773 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - e24c15c1-4692-4d15-a064-1d4850194518
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - aabec85d-8df3-4caa-9205-9a7477710bca
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - 07476198-0fb5-4af9-8eef-879c440c249b
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 8540599a-b39a-4f6e-b948-2fb3ddc1c24a
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - d17ebe9b-a444-4eec-96fb-583ab7b34f38
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 19567acf-71a3-43c4-99f3-2ee5145aa265
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - 5fc38936-49ab-4ca5-a8b4-a578f044f581
2014-10-14 08:51:34,774 [11] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,774 [8] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,774 [11] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 11
2014-10-14 08:51:34,774 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 9
2014-10-14 08:51:34,775 [9] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - 076b028c-1934-449a-ad17-1fc76ca3e5fb
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - 7a26c404-76a4-4435-9cd9-d322df61efb4
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e3bb78a5-8623-495e-98db-40e43c40d634
2014-10-14 08:51:34,774 [8] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,775 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - e38251e2-0916-48bf-bb9a-761fc1e0ac5e
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - e822c4a9-a307-4375-9d43-2fc06618867a
2014-10-14 08:51:34,776 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 545ea7df-77a4-44a0-8c9d-1a5755062255
2014-10-14 08:51:34,777 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - c2aff532-9e7e-42c9-9608-a8adfcb86dd0
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - de2c6395-5e6b-48db-b81e-e5c41521abb1
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - b84ef574-e7a3-4f16-a405-72cab7e6999c
2014-10-14 08:51:34,779 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 7be62279-00fa-442e-8137-651ae402dd75
2014-10-14 08:51:34,780 [9] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,780 [9] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 26b996bc-6685-4716-8237-cd3799b8a792
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 2834d4f4-0280-4c8d-ba62-2d491ca483c7
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - 70acb54e-5d71-40f1-98d8-039b5e05c1d2
2014-10-14 08:51:34,781 [9] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 9
2014-10-14 08:51:34,781 [11] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,781 [11] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,781 [9] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,781 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 14
2014-10-14 08:51:34,784 [14] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - b3673bf7-51f1-42a5-9395-5ef22b7c7fcf
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - e2da8b66-96f9-4ffb-8ce7-84521cfa8861
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 0ebcd0c1-52e2-4592-b026-9d19bce12b94
2014-10-14 08:51:34,784 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - a724690b-6ab1-4049-977c-80a1276e4239
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - 87518c99-7bdd-4e9b-8e6c-363fbe5c12d4
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - 58dbd214-f9ac-45e8-b0f1-9170a943a7cf
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - a87e0542-4993-4458-bb2c-132473835204
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 97f6e739-7ac0-41c9-8e62-089efd454bd3
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - c401faf6-0866-44e8-a9b2-0612b933808a
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - eb8e57a2-f73f-403e-a514-fe73710ca7c7
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 96f5c737-aad1-4774-9a2f-590678ac6e62
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 18c5670e-b27f-474c-8100-53962e3c7009
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - cb8327b9-b45c-451b-b936-393ebd41b1f8
2014-10-14 08:51:34,785 [14] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,785 [14] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 14
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Current bucket refCount# 1
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Removing bucket refCount# 0
2014-10-14 08:51:34,797 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Disposing on thread 13
2014-10-14 08:51:34,798 [13] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Unregistering observer default
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Configuration.ConfigContextBase - Disposing ConfigContext
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.134:11210
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.134:11210 - a29394e2-5b67-410a-8fa6-68a263a2351a
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.134:11210 - 554f7d15-4c08-485b-8235-6588e2cea61e
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - 6477b8a9-f68c-47da-b865-8fae30bdd3e9
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.134:11210 - d122c532-69c7-49a1-b867-ea5bc0f50cd5
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.135:11210
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.135:11210 - a5212957-d7b4-4e8a-8de2-c134e8b347cd
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.135:11210 - d54a3668-c9da-4e32-82ea-e451cfd3355e
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.135:11210 - 07dd9661-1f98-4964-9f47-a6882ae9968f
2014-10-14 08:51:34,798 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.121.136:11210
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.121.136:11210 - 54f98a31-fb83-4722-b7db-2d8c2620f131
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.121.136:11210 - d15a2d13-ae10-4280-a963-db384cfd6458
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.121.136:11210 - 94954350-41c8-4001-932b-02b683ca5c60
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Server - Disposing Server for 10.3.3.206:11210
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.Strategies.DefaultIOStrategy - Disposing DefaultIOStrategy for 10.3.3.206:11210 - 9d7480c8-a451-4b73-9cb2-75ae2dc0dacb
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionPool`1 - Disposing ConnectionPool for 10.3.3.206:11210 - 251020be-b0e7-489e-888d-05ff93bbf0ea
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.IO.ConnectionBase - Disposing connection for 10.3.3.206:11210 - d0a8de8e-89f8-46f5-b08c-1bd5f4982896
2014-10-14 08:51:34,799 [13] INFO Couchbase.Configuration.Server.Providers.ConfigProviderBase - Removing config for observer default
2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 13
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:37,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:39,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:42,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:44,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:47,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:49,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:52,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:54,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:57,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:51:59,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:02,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:04,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:07,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:09,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:12,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:14,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:17,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:19,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:22,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:24,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:27,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:29,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:32,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:37,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:39,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:42,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:44,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:47,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:49,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:52,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:54,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:57,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:52:59,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:02,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:04,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:07,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:09,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:12,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:14,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:17,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:19,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:22,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:24,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
2014-10-14 08:53:27,307 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False
....

 Comments   
Comment by Andrei Baranouski [ 14/Oct/14 ]
https://s3.amazonaws.com/bugdb/jira/NCBC-685/e82fc475/NCBC-685.zip
Comment by Jeff Morris [ 14/Oct/14 ]
It looks like the test harness is disposing the cluster?!:

2014-10-14 08:51:34,799 [13] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Attempting dispose on thread 13
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [14] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Cluster - Disposing Cluster
2014-10-14 08:51:34,806 [13] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - Disposing ConfigurationProvider: CarrierPublicationProvider
2014-10-14 08:51:34,807 [24] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.135:11210 - d01d19cb-d6b6-4531-a2cd-7ba68b2eba86. Trying again. - Disposed: False

Not sure why the ConnectionPool logger indicates the it's not disposed, though.
Comment by Jeff Morris [ 14/Oct/14 ]
>>seems like the problem is with view queries that are requested on rebalanced out nodes.

The connection pool is only used for memcached ops; views just use the server endpoints to construct the url in a round-robin fashion.
Comment by Andrei Baranouski [ 15/Oct/14 ]
Hi Jeff,

I'm able to reproduce it in command line on 172.23.107.172

stack trace there

  [External Code]
> Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 123 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Acquire() Line 126 C#
  Couchbase.dll!Couchbase.IO.ConnectionPool<Couchbase.IO.Strategies.EapConnection>.Couchbase.IO.IConnectionPool.Acquire() Line 178 C#
  Couchbase.dll!Couchbase.IO.Strategies.DefaultIOStrategy.Execute<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 40 C#
  Couchbase.dll!Couchbase.Core.Server.Send<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 90 C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry<string>(Couchbase.IO.Operations.IOperation<string> operation) Line 95 C#
  Couchbase.dll!Couchbase.Core.Buckets.CouchbaseBucket.Get<string>(string key) Line 967 C#
  Sdkd.dll!Sdkd.Commands.GetCommand.ExecuteSingle(string key, string value) Line 17 C#
  Sdkd.dll!Sdkd.Commands.KVCommand.Run() Line 24 C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.DispatchCommand(Sdkd.Protocol.Request req, Couchbase.Core.Buckets.CouchbaseBucket cli) Line 266 C#
  SdkdConsole.exe!SdkdConsole.Daemon.Handle.Run() Line 146 C#
  [External Code]
Comment by Jeff Morris [ 15/Oct/14 ]
Thanks Andrei. I know where this is happening, I just don't know most importantly WHY it's occurring. Basically, the connection pool is empty and all threads are waiting for a connection to become available. Since the PoolConfiguration.MaxSize has been reached, no more connections are being created. Note that there is a timeout for the max amount of time that a connection can be used, so either this isn't be honored (doubtful) or more like the connections are be destroyed somehow and not reused.

-Jeff
Comment by Jeff Morris [ 15/Oct/14 ]
This would cause a connection to not be re-added to the pool (since the auth failed on it while it was being initialized):

Line 1495903: 2014-10-15 14:47:44,782 [16] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire new: db69c7dd-2d91-4013-a086-5d739867e02b | 10.3.4.144:11210 | [0, 1] - 3dd67e96-c247-4870-aab1-d7d867426c4b - Disposed: False
Line 1495904: 2014-10-15 14:47:44,782 [16] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authenticating socket db69c7dd-2d91-4013-a086-5d739867e02b
Line 1495928: 2014-10-15 14:47:47,283 [16] DEBUG Couchbase.Authentication.SASL.CramMd5Mechanism - Authentication for socket db69c7dd-2d91-4013-a086-5d739867e02b failed: Operation timed out: the timeout can be configured by changing the PoolConfiguration.OperationTimeout property. The default is 2500ms.
Comment by Jeff Morris [ 15/Oct/14 ]
Patch: http://review.couchbase.org/#/c/42194/
Comment by Andrei Baranouski [ 16/Oct/14 ]
good results for the test and it doesn't stuck)

http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2Out-HYBRID/10-16-14/032835/183bac130ab3cd60442dfdcc92f9b61e-MC.html


but there are some unexpected statistics in other tests. will summarize them soon
Comment by Andrei Baranouski [ 16/Oct/14 ]
for some cases, for exmaple Rb2OutEpt I still see considerably smaller ops/sec view requests values during and after rebalance out
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-16-14/032248/0fc0dcd7fc34e7ad9aa92e6615c36329-CB.html
and MEMD:GENERIC error afte simple rebalance out 4-2 nodes
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2OutEpt-HYBRID/10-16-14/032248/0fc0dcd7fc34e7ad9aa92e6615c36329-MC.html

latency increased after rebalance Rb2In, it should be less than
http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/Rb2In-HYBRID/10-16-14/034067/f32ec4d1e53a4024e529676904f6d216-MC.html


MEMD:GENERIC error afte failover 1 EPT node and rebalance out
 http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/FoEptRb-HYBRID/10-16-14/034611/981225e9aa00099891b40ae02ad922bb-MC.html


results are from http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/35/consoleFull

Comment by Jeff Morris [ 17/Oct/14 ]
These are run off of my machine using SDKD (see attachements above). Overall the results looks significantly better: MEMD:GENERIC and EVBUCKET go down after rebound along with latency, while generally performance improves.
Comment by Andrei Baranouski [ 18/Oct/14 ]
Hi Jeff, I think we can close this ticket because the original problem has been fixed
and to keep track of all the inconsistencies / problems that we see in different runs VS envs I created a separate ticket NCBC-690
Comment by Jeff Morris [ 18/Oct/14 ]
Ok, sounds good.




Generated at Tue Oct 21 10:16:11 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.