[JCBC-450] 1.4.0 is not published on maven central Created: 17/Apr/14  Updated: 21/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 1.4.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Just reported by support, 1.4.0 doesn't seem to be on maven central.

 Comments   
Comment by Michael Nitschinger [ 18/Apr/14 ]
It shows up:

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.couchbase.client%22%20AND%20a%3A%22couchbase-client%22

and the artifacts can be downloaded.
Comment by Michael Nitschinger [ 18/Apr/14 ]
Dont think this is a bug, maybe the person needs to update his maven index in the IDE from time to time so it gets picked up properly by autocomplete.
Comment by Matt Ingenthron [ 21/Apr/14 ]
We determined that the issue is the lexical sort of the artifacts. In the future, we'll need to be more careful about how we publish DPs. I think the -SNAPSHOT approach understood by maven needs to be implemented. This should be fixed with a 1.4.1 shortly.




[JCBC-399] ViewFuture Listeners get completed twice Created: 11/Jan/14  Updated: 14/Jan/14  Resolved: 14/Jan/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.3.0
Fix Version/s: 1.3.1
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This is a regression from 1.3.0.

See https://github.com/ReactiveCouchbase/ReactiveCouchbase-core/issues/3 for more details

 Comments   
Comment by Michael Nitschinger [ 13/Jan/14 ]
http://review.couchbase.com/#/c/32305/




[JCBC-334] release notes for 1.2.0 Created: 24/Jul/13  Updated: 17/Sep/13  Resolved: 17/Sep/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 1.2
Security Level: Public

Type: Task Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Need published release notes for 1.2.0.

This would be expected around first week of September.

 Comments   
Comment by kzeller [ 24/Jul/13 ]
Important: as of 7/24 we are waiting on Ray to get ubu-1701 running. Java 1.2 files exist but are not yet publishable on the web.
Comment by Matt Ingenthron [ 29/Jul/13 ]
Update from Karen last week, the VM has been recovered. I believe we're good to go on docs.




[JCBC-360] Prepare for move to maven central Created: 10/Sep/13  Updated: 17/Sep/13  Resolved: 17/Sep/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.2
Security Level: Public

Type: Task Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-349] Calling flush() on CouchbaseClient throws a ClassCastException Created: 26/Aug/13  Updated: 04/Sep/13  Resolved: 04/Sep/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.9
Fix Version/s: 1.2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Alan Kleiman Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When calling flush() on CouchbaseClient a ClassCastException is thrown:

java.lang.ClassCastException: com.couchbase.client.CouchbaseMemcachedConnection cannot be cast to com.couchbase.client.CouchbaseConnection
at com.couchbase.client.CouchbaseClient.flush(CouchbaseClient.java:2421) ~[couchbase-client-1.1.9.jar:1.1.9]
at com.couchbase.client.CouchbaseClient.flush(CouchbaseClient.java:2410) ~[couchbase-client-1.1.9.jar:1.1.9]

The problem seems to be in CouchbaseClient:2421, maybe related to JCBC-323?

 Comments   
Comment by Michael Nitschinger [ 26/Aug/13 ]
Jup looks like a bug! Expect a fix for the next release.
Comment by Michael Nitschinger [ 27/Aug/13 ]
http://review.couchbase.org/#/c/28599/




[JCBC-339] Couchbase java client ignores provided username and always use bucket as username during HTTP authorization step Created: 31/Jul/13  Updated: 31/Jul/13  Resolved: 31/Jul/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.8
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Sergey Bushik Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Server: CentOS 2.6.32 x86_64, Client: MacOS Darwin 12.4.0, Java 1.6


 Description   
new CouchbaseClient(servers, "default", "username", "password") doesn't honor provided username and always use bucket as username during HTTP authorization step.

Outgoing request is:
FINE: www.MessageHeader@658f73867'>sun.net.www.MessageHeader@658f73867 pairs: {GET /pools HTTP/1.1: null}{Accept: application/json}{user-agent: Couchbase Java Client}{X-memcachekv-Store-Client-Specification-Version: 1.0}{Authorization: Basic ZGVmYXVsdDpwYXNzd29yZA==}{Host: 192.168.1.79:8091}{Connection: keep-alive}
Text value of Authorization header corresponds to "Basic default:password", while it should be "Basic username:password"

Therefore 401 unauthorized response received:
www.MessageHeader@92f1bf07'>sun.net.www.MessageHeader@92f1bf07 pairs: {null: HTTP/1.1 401 Unauthorized}{WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"}{Server: Couchbase Server 2.1.0-718-rel-enterprise}{Pragma: no-cache}{Date: Wed, 31 Jul 2013 13:10:21 GMT}{Content-Length: 0}{Cache-Control: no-cache}

The actual bug hides at line #131 of method com.couchbase.client.CouchbaseConnectionFactoryBuilder.buildCouchbaseConnection(final List<URI> baseList, final String bucketName, final String usr, final String pwd), wher usr parameter is not used anyhow and just ignored

Or see it at:
https://github.com/couchbase/couchbase-java-client/blob/master/src/main/java/com/couchbase/client/CouchbaseConnectionFactoryBuilder.java#L131

 Comments   
Comment by Michael Nitschinger [ 31/Jul/13 ]
Hi Sergey,

that's intended and not an issue. This is because the "user" is currently not being used, you always need to use the bucket name as the user and the password.

The only occasion where you dont get around with bucket level credentials is by adding/removing buckets, with the ClusterManager, and there you need to provide the admin creds.

Makes sense?
Comment by Sergey Bushik [ 31/Jul/13 ]
Technically, I see extended signature which accepts bucket, username & password parameters. Even though the credentials are valid, it throws out error, which says nothing about username parameter usage, but...

java.io.IOException: Server returned HTTP response code: 401 for URL: http://192.168.1.79:8091/pools
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readToString(ConfigurationProviderHTTP.java:424)
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readPools(ConfigurationProviderHTTP.java:210)
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:147)
at com.couchbase.client.CouchbaseConnectionFactory.getVBucketConfig(CouchbaseConnectionFactory.java:229)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:241)
Exception in thread "main" com.couchbase.client.vbucket.ConfigurationException: Configuration for bucket "default" was not found in server list ([http://192.168.1.79:8091/pools]).

That's confusing. By the way, I debugged and found that it's possible to authenticate using bucket/username/password over HTTP (modified ConfigurationProviderHTTP).




[JCBC-278] ConnectionFactoryBuilder has wrong default settings Created: 02/Apr/13  Updated: 03/Apr/13  Resolved: 03/Apr/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.4
Fix Version/s: 1.1.5
Security Level: Public

Type: Task Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The Builder uses wrong failure mode and default hash, they differ from instantiating the factory directly. This causes inconsistencies and also EPT node failures not to be detected properly.

 Comments   
Comment by Frank Weigel [ 03/Apr/13 ]
WIll changing hash affect customers upgrading? (i.e. not finding old data)




[JCBC-261] Warmup Backoff breaks Memcache Buckets Created: 11/Mar/13  Updated: 12/Mar/13  Resolved: 12/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.3
Fix Version/s: 1.1.4
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
A recent change regarding warmup backoff breaks compat with memcache type buckets.




[JCBC-251] Observe ignores replica on index 0 Created: 24/Feb/13  Updated: 27/Feb/13  Resolved: 27/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0, 1.1.1, 1.1.2
Fix Version/s: 1.1.3
Security Level: Public

Type: Task Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
A replica on index 0 is not added to the broadcast observe list and therefore leads to timeouts when persistence constraints are provided.




[JCBC-245] Docs: Javadoc states that a java.lang.String is required for the value in a write operation, whereas online docs state object Created: 12/Feb/13  Updated: 27/Feb/13  Resolved: 27/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.2
Fix Version/s: 1.1.3
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Java docs here: http://www.couchbase.com/autodocs/couchbase-java-client-1.1.2/

All set/replace/add/cas operations state something like:
cas(java.lang.String key, long cas, java.lang.String value, PersistTo req)

Whereas the docs here: http://www.couchbase.com/docs/couchbase-sdk-java-1.1/api-reference-summary.html

State that it's an "Object value"

Looking deeper, it appears that the memcached methods inherited from spy use java.lang.object which leads to user confusion when trying to compare the two.

 Comments   
Comment by Perry Krug [ 12/Feb/13 ]
Adding a library component since the Java docs show what the required type is, and if we are limiting to strings, we are preventing people from using these methods with non-string objects without writing their own transcoder which seems a little arduous in the simpler cases.
Comment by Michael Nitschinger [ 19/Feb/13 ]
That's purely a lib bug, since those should also require Object instead of String.




[JCBC-242] The flush() operations always fails with "401 Unathorized" message Created: 08/Feb/13  Updated: 08/Feb/13  Resolved: 08/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.2
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Balint Ureczky Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: 1m
Time Spent: Not Specified
Original Estimate: 1m
Environment: uname -a
Linux Mint 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

java -version
java version "1.7.0_11"
Java(TM) SE Runtime Environment (build 1.7.0_11-b21)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)

Java couchbase client library
1.1.2 taken from: http://files.couchbase.com/maven2/couchbase/couchbase-client/1.1.2/couchbase-client-1.1.2.jar

Attachments: Java Source File CouchbaseMagic.java    
Issue Links:
Duplicate
duplicates JCBC-173 flush will not work owing to MB-7381 Resolved

 Description   
The flush operations seems to require some kind of authorization, however the client library does not provide a way to do it.

The documentation is does not mention anything about this.

Is there a magical way I can call flush?
I attached the source code which reproduces the problem.

 Comments   
Comment by Michael Nitschinger [ 08/Feb/13 ]
Note that this is a known limitation because of the CouchbaseServer

See the linked issue for more information!
Comment by Michael Nitschinger [ 08/Feb/13 ]
Note that this is marked to be fixed in 2.0.1 which is the next bugfix release of couchbase server. Once upgraded, it should "magically" work!




[JCBC-241] Paginator object doesn't move onto next "page" correctly Created: 07/Feb/13  Updated: 08/May/13  Resolved: 08/May/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.2
Fix Version/s: 1.1.6
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File paginator-java.diff    
Issue Links:
Dependency

 Description   
Customer found his own issue:
I had a look at the source code for the java driver and found the error there. It is in:

com.couchbase.client.protocol.views.Paginator.getNextPage()

Right after calculating the number of "remaining" keys, it sets the limit of the query to the remaining number of keys and re-runs the query. What is missing here is:

    q.setSkip(totalDocs);

To actually start from the entry after the last entry on the current page. I have tried this and it works. But there is more code in this method and it is not very easy to follow so important to make sure that this fits.


I am enclosing a diff.

 Comments   
Comment by Perry Krug [ 07/Feb/13 ]
Update, but the bug still stands as described.

It seems that my fix is not the correct one. The bug seem to be that although the Paginator specifies setStartkeyDocID, the query doesn't start from this key.

I have debugged the Paginator class – and when resolving the next page it instructs the query to start from a specific doc id, e.g:

Query: ?limit=50&startkey=144&skip=0&stale=false&startkey_docid=144

But as a result of this query I get:

Got key: 1
Got key: 10
Got key: 100
Got key: 101
Got key: 102
Got key: 103

Got key: 141
Got key: 142
Got key: 143

Every time. So the bug is in the Query context somewhere.

Also – a note of inefficiency in the Paginator class: why perform two queries every time? One for the content of the "page" and one to find the row for the next page? Would be better to query for page + 1 and keep the last separate from the ViewResponse.

So, what I can conclude is that when the Paginator bug is resolved it should work fine for us. The additional query for the last row could be an issue though, since it uses setSkip(). So could this be treated as a bug as well?




[JCBC-219] Fix reconnect logic when server closes the socket. Created: 24/Jan/13  Updated: 29/Jan/13  Resolved: 29/Jan/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0, 1.1.1
Fix Version/s: 1.1.2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When the server disconnects on us during a valid remove/failover rebalance, it is the case that the CouchbaseClient stays connected and doesnt try to disconnect and retry. This leaves the client in a dangerous state where no updates are fetched and therefore the map is incorrect.

 Comments   
Comment by Michael Nitschinger [ 25/Jan/13 ]
http://review.couchbase.org/#/c/24182/




[JCBC-168] CouchClient.getView always throws an exception Created: 05/Dec/12  Updated: 11/Dec/12  Resolved: 11/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1-beta
Fix Version/s: 1.1.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Chris Tashjian Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
It would seem that client.getView(String designdocName, String viewName) has stopped working in 1.1-beta. It now throws:

java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:66)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:492)

This was working fine in 1.1-dp4.

 Comments   
Comment by Michael Nitschinger [ 06/Dec/12 ]
Hi Chris,

thanks for filing this. There has been a change that makes the ViewTimeout tunable, which I think may be the problem here. All my tests go through without problems, so I think you're using the FactoryBuilder right?

Can you please give me your bootstrap code and all the timeouts you are using? Also, please check your boot logs if it says something about a low view timeout. Thanks!
Comment by Chris Tashjian [ 06/Dec/12 ]
We're using the default ViewTimeout...

CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
builder.setAuthDescriptor(new AuthDescriptor(new String[]{"PLAIN"}, new PlainCallbackHandler(bucketName, password)));
builder.setOpTimeout(opTimeout);
if (failureMode != null) {
    builder.setFailureMode(FailureMode.valueOf(failureMode));
}
CouchbaseClient client = createClient(builder, uris);
Comment by Chris Tashjian [ 06/Dec/12 ]
Ok, I think I was able to get this to work by adding "builder.setViewTimeout(5000);".

At one point I had it set to 3000 and got a timeout exception... however, if you don't explicitly call setViewTimeout, it seems that you get the less informative RuntimeException that I originally filed this for. It might be helpful if the constructor for CouchbaseConnectionFactoryBuilder set some kind of default timeout.
Comment by Matt Ingenthron [ 11/Dec/12 ]
http://review.couchbase.org/#/c/23189/
Comment by Michael Nitschinger [ 11/Dec/12 ]
fixed and pushed to master!




[JCBC-162] re-enable delete observe against 2.0 server Created: 03/Dec/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1-dp3, 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Matt Ingenthron Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
At one point, the observe delete had been disabled on the master branch owing to changes in approach and needing to get a change merged for delete. Once this is complete, we need to revert the change.

See the review for the remove here: http://review.couchbase.org/20918

 Comments   
Comment by Michael Nitschinger [ 03/Dec/12 ]
implemented and pushed to master, will be available in beta.




[JCBC-160] merge in changes from the release10 branch for 1.1beta Created: 03/Dec/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In particular, the fix for JCBC-70 is needed, but there are a couple of changes which need to be merged in

 Comments   
Comment by Michael Nitschinger [ 03/Dec/12 ]
merged, will be available in the beta release.




[JCBC-158] add a debug=true option to view query options Created: 29/Nov/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
To be able to diagnose issues that may come up with views, please add a debug=true query parameter. Optionally, we could add a cbclient.properties parameter to be able to turn on view debugging (for all view requests or just some that match a regex?) without code changes (though, with restarts).

 Comments   
Comment by Michael Nitschinger [ 30/Nov/12 ]
http://review.couchbase.org/#/c/22923
Comment by Michael Nitschinger [ 03/Dec/12 ]
pushed to master, will be available in the beta relase.




[JCBC-154] merge in changes from the release11c branch Created: 26/Nov/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When reviewing some changes on the "c" branch, I happened to notice there may be one change that hasn't been submodule merged

Upon further inspection, it looks like a few other changes haven't been merged either. We'll need to merge this branch back in.

 Comments   
Comment by Michael Nitschinger [ 29/Nov/12 ]
You already did the work, so assigning it to you.
Comment by Matt Ingenthron [ 03/Dec/12 ]
http://review.couchbase.org/#/c/22973/
Comment by Michael Nitschinger [ 03/Dec/12 ]
merged into master, available in beta.




[JCBC-153] Raise view timeout from 60s to 75s Created: 21/Nov/12  Updated: 03/Dec/12  Resolved: 27/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Improvement Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Since the server side view timeout is 60s, we should raise the client side a bit higher. 75s seems like the right number. It should be tuneable though.

 Comments   
Comment by Michael Nitschinger [ 22/Nov/12 ]
http://review.couchbase.org/#/c/22755/
Comment by Michael Nitschinger [ 27/Nov/12 ]
Pushed to master, will be available in dp5/beta.




[JCBC-150] "Failed to access the view" error when querying a view with reduce Created: 19/Nov/12  Updated: 27/Nov/12  Resolved: 27/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Chris Tashjian Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Attempting to make a call to CouchbaseClient.query(view, query) with a view that has a reduce (even just "_count") results in the following exception when results are returned:

java.lang.RuntimeException: Failed to access the view
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:634)

Worth noting:
- If the query returns no results (ViewResponse.size() returns 0), you do not get the exception...
- If the reduce method is removed from the view, everything works fine. However, using query.setReduce(false) does not work.



 Comments   
Comment by Michael Nitschinger [ 20/Nov/12 ]
Hi Chris,

thanks for the report. I'll investigate and let you know!

Thanks,
Michael
Comment by Michael Nitschinger [ 21/Nov/12 ]
Please make sure to use setReduce(true) and .setReduce(false) explicitely on a view with a reduce function right now.

The following code works with setReduce(true) and false, but will throw the exception you mentioned when setReduce is not used at all:

    query = new Query();
    //query.setReduce(false);
    view = client.getView(DESIGN_DOC_W_REDUCE, VIEW_NAME_W_REDUCE);
    reduce = client.query(view, query);
    itr = reduce.iterator();
    while(itr.hasNext()) {
      ViewRow row = itr.next();
      System.out.println(row.getKey());
    }

Comment by Michael Nitschinger [ 21/Nov/12 ]
After more investigation, this is really a shortcoming and should be handled in this changeset:

http://review.couchbase.org/#/c/22710/

It makes sure that reduce = true when nothing is set and the view contains a reduce function. The exception was misleading because it covered a JSON parsing bug underneath.
Comment by Michael Nitschinger [ 27/Nov/12 ]
fixed and pushed to master; will be available in dp5/beta.




[JCBC-147] Rename getViews() to getDesignDocument() Created: 14/Nov/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Improvement Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
blocks JCBC-63 add APIs for creating and deleting de... Resolved

 Description   
The getViews() method should be renamed to getDesignDocument() before the API is stable. This is also in preparation for the createDesignDocument() and deleteDesignDocument() additions.

 Comments   
Comment by Michael Nitschinger [ 21/Nov/12 ]
http://review.couchbase.com/#/c/22713/
Comment by Matt Ingenthron [ 03/Dec/12 ]
This breaks API over DP, and should be release noted.
Comment by Michael Nitschinger [ 03/Dec/12 ]
fixed and merged into master, will be available in beta!




[JCBC-144] flush command needs to use RESTful flush for Couchbase Buckets Created: 11/Nov/12  Updated: 11/Dec/12  Resolved: 11/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.1.0
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Matt Ingenthron Assignee: Matt Ingenthron
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The current flush command needs to be connected to the RESTful flush.

 Comments   
Comment by Michael Nitschinger [ 12/Nov/12 ]
To my findings there are two ways to implement this feature:

- Reuse the BucketManager for this, since it already provides basic capabilities for flushing (but needs some extension).
- Reimplement the whole thing.

I would go with extending the BucketManager for this, since it would also keep the CouchbaseClient itself lean.

There is one thing that we need to decide upon: we can't just override the flush() method, because the returned value is different (we can't return an operation future with the current implementation). If we want to, we could "fake" it into one (but I doubt it makes sense). Therefore I propose a new method (flushBucket) which should be used with the couchbase client and we need to document that the old flush methods only work against memcached servers.
Comment by Michael Nitschinger [ 12/Nov/12 ]
Tracked here: http://review.couchbase.com/#/c/22445/
Comment by Matt Ingenthron [ 03/Dec/12 ]
Pushing out to release 1.1.0, as it's secondary functionality.
Comment by Matt Ingenthron [ 07/Dec/12 ]
http://review.couchbase.org/#/c/22445/




[JCBC-110] observe method does not correctly check persistence or replication status on replica node/vbuckets Created: 12/Sep/12  Updated: 16/Sep/12  Resolved: 16/Sep/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1dp2
Fix Version/s: 1.1-dp3
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When performing some additional testing, Abhinav noticed some unusual behavior in interacting with clients. After checking with Mike a bit, they noticed that observe protocol were not going to replica node/vbuckets during the poll interval loop.

 Comments   
Comment by Matt Ingenthron [ 12/Sep/12 ]
Assigning to Mike to update/replace the description.
Comment by Mike Wiederhold [ 16/Sep/12 ]
http://review.couchbase.org/#/c/20847/
http://review.couchbase.org/#/c/20848/
http://review.couchbase.org/#/c/20849/
http://review.couchbase.org/#/c/20850/
http://review.couchbase.org/#/c/20851/
http://review.couchbase.org/#/c/20852/




[JCBC-47] Query.copy does not copy includeDocs property Created: 12/May/12  Updated: 01/Jun/12  Resolved: 01/Jun/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1dp
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Steven Cooke Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: all java


 Description   
Query.java line 200 in function copy():
setIncludeDocs(willIncludeDocs());

Should be
query.setIncludeDocs(willIncludeDocs());

Paginated queries depend on copy() so paginatedQuery requiring the document fail with:
 java.lang.UnsupportedOperationException: This view result doesn't contain documents

 Comments   
Comment by Steven Cooke [ 12/May/12 ]
Perhaps a better solution:

  /**
   * All values in args map must be immutable
   */
  private Query(Query src) {
    args = new HashMap<String, Object>(src.args);
    includedocs = src.willIncludeDocs();
  }

  public Query copy() {
    return new Query(this);
  }




[JCBC-40] paginatedQuery throws NoSuchElementException and NPE Created: 29/Apr/12  Updated: 03/Dec/12  Resolved: 27/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1dp
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Steven Cooke Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: java


 Description   
view iteration is incomplete and throws exceptions for
groovy code:
paginator = client.paginatedQuery(view, query, n)
while (paginator.hasNext()) {
  row = paginator.next() // Exception on this line
  ...
}


java.util.NoSuchElementException
        at java.util.LinkedList$ListItr.next(LinkedList.java:698)
        at com.couchbase.client.protocol.views.Paginator.next(Paginator.java:76)
        at com.couchbase.client.protocol.views.Paginator.next(Paginator.java:35)
        at java_util_Iterator$next.call(Unknown Source)

AND

java.lang.NullPointerException
        at com.couchbase.client.protocol.views.Paginator.getNextPage(Paginator.java:93)
        at com.couchbase.client.protocol.views.Paginator.hasNext(Paginator.java:67)
        at java_util_Iterator$hasNext.call(Unknown Source)

Paginator also has a couple odd returns
1) next() can return null, but it should never return null unless lastRow is null. I don't think lastRow should ever be null.
2) getNextPage() has a return type, HttpFuture<ViewResponse>, but always returns null


 Comments   
Comment by Steven Cooke [ 15/May/12 ]
some workarounds for NPE in getNextPage():

1) Check for a null ViewResponse in getNextPage and return the ViewResponse instead of a future or null. The hasNext method can then check the value of getNextPage returning false if there is no ViewResponse.

 2) to bypass isJsonObject for numeric keys in Query.getArgs:
boolean numericKey = (key.equals(STARTKEY) && value.toString().matches("\\d+")); // still not json number but catches most common case of positive
integer keys if (key.equals(STARTKEYDOCID) || numericKey ) { return key + "=" + value;

 3) Skip parameter needs to be zero for pages after the first. In Paginator:
 public boolean hasNext() {
   if (!pageItr.hasNext() && page.size() < docsPerPage) {
     return false;
   } else if (!(rowsIterated < docsPerPage)) {
     lastRow = pageItr.next();
     query.setSkip(0);
     query.setStartkeyDocID(lastRow.getId());
     query.setRangeStart(lastRow.getKey());
     return (getNextPage(query) != null);
   }
   return true;
}
Comment by Steven Cooke [ 15/May/12 ]
Note: #2 fails for numeric keys that are expected to be strings, an issue raised in SPY-62
Comment by Raghavan Srinivas (Inactive) [ 16/Sep/12 ]
I have revamped the paginator and the code to iterate over page and within page looks something like this.

       while (result.hasNext()) {
           ViewResponse response = result.next();
           for (ViewRow row: response) {
             System.out.println("Inner loop " + row.getId());
           }
           System.out.println("<===Outer loop====>");
       }
  
Comment by Matt Ingenthron [ 18/Sep/12 ]
http://review.couchbase.org/#/c/20898/
Comment by Matt Ingenthron [ 18/Sep/12 ]
After discussion with Rags, there are some scenarios not as well tested as we'd like and there was some clarification on how this is expected to work.
Comment by Raghavan Srinivas (Inactive) [ 26/Sep/12 ]
This revolves around getKey() returning a string which is not the case always. It could return other types (such as int) and this will affect the way PaginatedQuery and other queries behave.
Comment by Matt Ingenthron [ 06/Oct/12 ]
In an email, Steven Cooke commented:

A couple other things I noticed:

The fix breaks all the original test cases as return type of next is no longer the same. If the API is intended to be non-breaking, the
 test cases and next should be reverted.

The private method
getNextPage has a return value that is never used. This looks fishy
Comment by Matt Ingenthron [ 06/Oct/12 ]
This isn't intended to be API compatible actually. Rags and I talked through it and decided that the odd returns of nulls were odd. I don't know that the API you see here is final either.

We would appreciate any additional feedback on this API or how you think it should work.

Once we hit beta, we'll keep the API stable.
Comment by Michael Nitschinger [ 14/Nov/12 ]
http://review.couchbase.org/#/c/22513/
Comment by Michael Nitschinger [ 27/Nov/12 ]
Fixed and pushed to master. Will be available in dp5/beta.




[JCBC-35] Exception during reconfiguring of client Created: 23/Apr/12  Updated: 26/Mar/13  Resolved: 26/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.0.2
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Marcus Nylander Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: couchbase-client 1.0.2

java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Linux 2.6.18-238.19.1.el5 #1 SMP Sun Jul 10 08:43:41 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Attachments: Text File log.txt    

 Description   
Multiple different stackstraces below also see the following right before it starts to fail:

"Node exepcted to receive data is inactive. This could be due to afailure within the cluster. Will check for updated configuration. Key without a configured node is: akey12324234"

Then it's fails a number of times and

Failed to reconfigure client, staying with previous configuration.
java.lang.IllegalArgumentException: TODO: refactor this
        at com.couchbase.client.vbucket.config.CacheConfig.getVbucketsCount(CacheConfig.java:55)
        at com.couchbase.client.vbucket.config.DefaultConfig.compareTo(DefaultConfig.java:145)
        at com.couchbase.client.vbucket.VBucketNodeLocator.updateLocator(VBucketNodeLocator.java:133)
        at com.couchbase.client.CouchbaseConnection.reconfigure(CouchbaseConnection.java:120)
        at com.couchbase.client.CouchbaseClient.reconfigure(CouchbaseClient.java:176)
        at com.couchbase.client.vbucket.ReconfigurableObserver.update(ReconfigurableObserver.java:54)
        at java.util.Observable.notifyObservers(Observable.java:142)
        at com.couchbase.client.vbucket.BucketMonitor.setBucket(BucketMonitor.java:257)
        at com.couchbase.client.vbucket.BucketMonitor.startMonitor(BucketMonitor.java:187)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:243)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.finishResubscribe(ConfigurationProviderHTTP.java:215)
        at com.couchbase.client.CouchbaseConnectionFactory.checkConfigUpdate(CouchbaseConnectionFactory.java:182)
        at com.couchbase.client.CouchbaseConnection.addOperation(CouchbaseConnection.java:165)
        at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:639)
        at net.spy.memcached.MemcachedClient.asyncGet(MemcachedClient.java:835)
        at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:997)




 Failed to reconfigure client, staying with previous configuration.
java.lang.IllegalArgumentException: TODO: refactor this
        at com.couchbase.client.vbucket.config.CacheConfig.getVbucketsCount(CacheConfig.java:55)
        at com.couchbase.client.vbucket.config.DefaultConfig.compareTo(DefaultConfig.java:145)
        at com.couchbase.client.vbucket.VBucketNodeLocator.updateLocator(VBucketNodeLocator.java:133)
        at com.couchbase.client.CouchbaseConnection.reconfigure(CouchbaseConnection.java:120)
        at com.couchbase.client.CouchbaseClient.reconfigure(CouchbaseClient.java:176)
        at com.couchbase.client.vbucket.ReconfigurableObserver.update(ReconfigurableObserver.java:54)
        at java.util.Observable.notifyObservers(Observable.java:142)
        at com.couchbase.client.vbucket.BucketMonitor.setBucket(BucketMonitor.java:257)
        at com.couchbase.client.vbucket.BucketMonitor.replaceConfig(BucketMonitor.java:307)
        at com.couchbase.client.vbucket.BucketUpdateResponseHandler.messageReceived(BucketUpdateResponseHandler.java:81)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at com.couchbase.client.vbucket.BucketUpdateResponseHandler.handleUpstream(BucketUpdateResponseHandler.java:194)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:513)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:497)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:434)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:340)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:272)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


 Comments   
Comment by Matt Ingenthron [ 11/Oct/12 ]
This may have been fixed in configuraton changes in 1.0.3, but that should be verified.
Comment by Michael Nitschinger [ 28/Nov/12 ]
http://review.couchbase.org/#/c/22878/
Comment by Michael Nitschinger [ 29/Nov/12 ]
fixed and pushed to master, will be available in in dp5/beta.
Comment by Michael Nitschinger [ 26/Mar/13 ]
Already fixes back in time, don't know why it is open.




[JCBC-27] race condition during startup Created: 23/Mar/12  Updated: 04/Mar/13  Resolved: 04/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.0, 1.0.1, 1.1dp, 1.1.0
Fix Version/s: 1.1.3
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
During startup, if there is no authentication (and thus no authentication latch) we can reply with errors before we get the configuration back and settled in with the node locator. This should be more reliable.

 Comments   
Comment by Matt Ingenthron [ 23/Mar/12 ]
See http://www.couchbase.com/forums/thread/fast-computer-race-condition-java-client
Comment by Matt Ingenthron [ 29/Aug/12 ]
I think part of the solution on this is to poll the configuration until it transitions from warmup to healthy.
Comment by Matt Ingenthron [ 08/Oct/12 ]
The idea is that there is a section of the code that walks the URIs, finds the bucket, then after finding it sets up the stream for the configuration. When it first finds the bucket, if it's in a "warmup" state, (easy to simulate by restarting a server) it will show that it is and will not have a vbucket map. At that point, we should loop without setting up the stream *or* we should set up the stream and let anything handling reconfigure handle the transition from warmup to warmed up.
Comment by Michael Nitschinger [ 30/Nov/12 ]
http://review.couchbase.com/#/c/22933/1
Comment by Matt Ingenthron [ 03/Dec/12 ]
Still working out the flow here. Based on our current understanding, this can be deferred to 1.1.0 or even post since it's an enhancement for reliable operation in a secondary or tertiary circumstance. Should be release noted though.
Comment by Matt Ingenthron [ 27/Feb/13 ]
Determined that the proposed approach is a good change, but better change is needed. That's tracked under JCBC-255.




[JCBC-21] Move the view code into CouchbaseClient Created: 23/Jan/12  Updated: 21/Mar/12  Resolved: 21/Mar/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.0
Security Level: Public

Type: Task Priority: Blocker
Reporter: Mike Wiederhold Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-20] CouchbaseClient instances consume all CPU resources Created: 10/Nov/11  Updated: 29/Jun/12  Resolved: 29/Jun/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1dp2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Kurtis Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Win7, centOS 5.4, tomcat 6.x, java 1.6.x

Attachments: Zip Archive CouchbaseTest.zip     PNG File resource_usage.png     Java Source File ViewConnection.java    

 Description   
When creating one or more CouchbaseClient instances, each one appears to consume 100% of the available CPU resources per core, at least on machines I've tested.
Test case (eclipse project and ant build) included.
Also included a snapshot of my system graph on my dev machine while running the included test.

 Comments   
Comment by Kurtis [ 10/Nov/11 ]
Also relevant are, this issue at google code:
http://code.google.com/p/spymemcached/issues/detail?id=218

And this thread in the Couchbase forums:
http://www.couchbase.org/forums/thread/using-couchbaseclient-connect
Comment by Matt Ingenthron [ 10/Nov/11 ]
I've reproduced this issue.

After investigating it a bit, I've found that simplifying the test to use either MemcachedClient or MembaseClient doesn't demonstrate the same issue. Under both of those, CPU usage is modest.

Further, more profiling seemed to show time on MacOS off in kqueue poll from the CouchbaseNode's run() method.

I suspect there's a thread safety issue here causing badness.

Asking Mike to review for any possible thread safety issues when there are multiple CouchbaseClients running.
Comment by Mike Wiederhold [ 30/Nov/11 ]
Were currently moving a few things around in the client. No user facing API changes, but still moving this around. I'll make sure to get this fixed as soon as possible though.
Comment by Alan Wood [ 04/May/12 ]
I've put a possible fix as an attachment to this issue. It works for us, and doesn't block shutdown, other nodes, etc.

The file was also submitted here (as an alternative to the ViewNode patch placed there earlier):
http://www.couchbase.com/issues/browse/JCBC-26
Comment by Steven Cooke [ 15/May/12 ]
Is this still open? ViewConnection and CouchbaseConnection are both threads with tight run loops. The quick fix is to sleep(50) or so. Not sure what the real solution should be, but it seems any threaded connection to the server to get server state can be handled by a singleton.

e.g.

public void run() {
  while(true) {}
}

vs

public void run() {
  while (true) {
    sleep(50); // utility function to hide try/catch
  }
}

First will hose the CPU, second will not
     




[JCBC-70] Client fails to reconnect to server of non-default memcached bucket after failover and add back Created: 28/Jun/12  Updated: 30/Jan/13  Resolved: 30/Jan/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.0.3
Fix Version/s: 1.1.2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GZip Archive sc_1_plabq11.dev.sabre.com.out.gz    
Issue Links:
Dependency
depends on SPY-102 Ensure all nodes are in the list of n... Resolved
Duplicate
is duplicated by SPY-111 Assertion/NPE Exception when trying t... Resolved

 Description   
In earlier tests with reconnecting to a node on failover we used default memcached bucket. But when we tested the same scenario with a non-default bucket, we noticed the client did not reconnect (due to a null pointer exception internally). I have attached the SDK logs for this scenario where we used "IndexByLniataData" memcached bucket. The problem presents when adding the node back after a failover.
  
11:34:43,411 DEBUG [Memcached IO over {MemcachedConnection to /10.14.5.119:11210}] [CouchbaseMemcachedConnection] Selecting with delay of 3038ms
Exception in thread "Thread-3" java.lang.NullPointerException
        at net.spy.memcached.auth.AuthThread.buildOperation(AuthThread.java:117)
        at net.spy.memcached.auth.AuthThread.run(AuthThread.java:86)

Logs/stack trace attached.

 Comments   
Comment by Matt Ingenthron [ 22/Aug/12 ]
I've spent a bit of time analyzing this issue, and it's not clear what the cause is. It is correct though that this would cause the auth thread to die, and as such authentication to the node would never complete.

There is a safeguard already in that the continuous timeout threshold will kick in and then the connection will be rebuilt. I don't know if this issue comes up all of the time, but assuming it's a rare event we'd see 1000 operations timeout (by default) followed by the connection being rebuilt.

We'd have to add some diagnostic information to the client and reliably reproduce this to identify the issue. I think the scenario is:
1) set up a cluster of say 3 nodes
2) configure a client, have it work with an authenticated memcached bucket on the cluster
3) faillover a node by clicking on "failover" in the console
4) add the node back by clicking on "add back"

Is this correct?
Comment by Perry Krug [ 23/Aug/12 ]
That appears correct. The customer has been able to reliably reproduce this, but since so much time has passed I would be hesitant in going back to them if not necessary...
Comment by Matt Ingenthron [ 09/Jan/13 ]
There is an open changeset for this. Please determine if it is correct, needs to go in.
Comment by Michael Nitschinger [ 30/Jan/13 ]
Duplicate of Spy-111




[JCBC-63] add APIs for creating and deleting design documents for CBS 2.0 Created: 08/Jun/12  Updated: 29/Nov/12  Resolved: 29/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1dp
Fix Version/s: 1.1-beta
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Dipti Borkar Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
depends on JCBC-147 Rename getViews() to getDesignDocument() Resolved

 Description   
There are have been several requests from users to have APIs to create and delete views from the Java (and other) SDKs.
This is a MUST have for some customers and needs to be a part of the Client SDK we release prior to Beta.

 Comments   
Comment by Matt Ingenthron [ 08/Jun/12 ]
I'm in agreement. One question is do we also want this API to be involved in the view materialization process. I think the answer is probably yes, but this can be complicated since it can potentially take a very long time with little feedback.
Comment by Dipti Borkar [ 08/Jun/12 ]
When you say View materialization process? what exactly do you mean? given that index building happens at query time, I don't think so.

However, there are other APIs that may be more interesting to have. explicitly triggering compaction, configuring min time duration / min # of mutations to trigger index building.
Comment by Matt Ingenthron [ 24/Jul/12 ]
What I mean by the view materialization process is that we, in the web console, have this flow where we advise people to take their "dev_aview" and run a full cluster dataset query on it before publishing it to production. This way, when it's published as "aview", the view will have been mostly materialized. It's a nice thing for updating a view.

The reason this is complicated is that it's an HTTP request that may go for a very long, long time. I can think of a way around it (poll it with a timeout until it's fast), but it's suboptimal unless we start querying view materialization progress.
Comment by Michael Nitschinger [ 26/Aug/12 ]
Matt, what if we wait for it in a separate thread and return with a future when it's done? We just need to make the user aware that it can take a long time, and if they add a view for development that they'd have to publish it separately.

Another idea would be to make the "let's do a full dataset query" optional when the view gets published since someone may want to do that later or from the web-ui.
Comment by Matt Ingenthron [ 26/Aug/12 ]
Michael: that's effectively what we do right now since it's the query against the view that takes a long time. The design document management operations do not. The view requests are not quite done asynchronously, that's true.

The main point I was raising was that the Web Console provides for functionality to execute a view before pushing it in production and workflow to take it from "dev_thing" to "thing" knowing that the view has been materialized. In the SDK, I don't think we'll try to replicate that workflow, but the same functionality is definitely available.
Comment by Michael Nitschinger [ 05/Oct/12 ]
Just to keep everyone updated, I started developing it here: http://review.couchbase.org/#/c/21380/
Comment by Michael Nitschinger [ 29/Nov/12 ]
this has finally been merged to master.




[JCBC-470] Implement Carrier-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-469] Implement HTTP-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-468] Implement EBNF-based N1QL DSL Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-457] Force carrier config fetching on reconnect of node. Created: 07/May/14  Updated: 12/May/14  Resolved: 08/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-454] Query View bug Created: 27/Apr/14  Updated: 05/May/14  Resolved: 05/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Nas Kavian Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File stack trace.png    
Issue Links:
Duplicate
duplicates SPY-163 Bulk latch never counts down on empty... Resolved

 Description   
1.3.2 works, but 1.4.0 is broken.

Perform a simple query where you set the key to something non-existent and the query will never return. I think an empty result set is properly returned from the server but the Java client doesn't handle it properly afterwards.

      final Query query = new Query();
      query.setIncludeDocs(true);
      query.setKey(ComplexKey.of("abc")); << Non-existent key
      query.setStale(Stale.FALSE);
      final ViewResponse response = client.query(view, query); << Doesn't return.

I think the attached stack trace gives some idea of where it's hung up.

 Comments   
Comment by Michael Nitschinger [ 05/May/14 ]
Hi,

thanks for reporting this - it is a known issue and is already fixed in master, will be in the next bugfix release 1.4.1 this week hopefully.




[JCBC-453] Also check for new configs with different failure modes. Created: 25/Apr/14  Updated: 08/May/14  Resolved: 08/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-449] Add an overview to the javadoc and the CouchbaseClient class Created: 15/Apr/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The HTML generated autodocs should have a better intro and get the user to the CouchbaseClient class more directly. For instance, if you go here:
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.0/index.html

One would have to click around for a bit to find what they're searching for.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
Are you sure this is possible with javadoc? I googled around a bit and didn't find anything.. well we could make it work with changing the files after generation or so.
Comment by Michael Nitschinger [ 17/Jun/14 ]
I fixed that in my build script, check out: http://www.couchbase.com/autodocs/couchbase-java-client-1.4.2/index.html




[JCBC-418] Deadlock in ViewConnection with Rebalance and retry requests Created: 21/Feb/14  Updated: 22/Feb/14  Resolved: 22/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.0-dp1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-413] Race condition in cluster manager class Created: 11/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Juan Manuel Musacchio Assignee: Michael Nitschinger
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I've been experiencing some connection issues ("Unable to connecto to cluster") while listing the couchbase buckets. After some debugging I realized that could be a race condition in the ClusterManager sendRequest method. The following are the involved lines (took it from master branch):

latch.countDown(); //line 474
success.set(true); //line 475
response.set(result); //line 476

The thing is that the mutex is decreased before setting sucess and response values, so later the code will check the success variable and could fail if success is not set before the thread starts (mutex is 0). I think the code should be:

success.set(true);
response.set(result);
latch.countDown();

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
Thanks, good catch. I'll get the fix in 1.4.2
Comment by Michael Nitschinger [ 02/Jun/14 ]
http://review.couchbase.org/#/c/37719/1
Comment by Michael Nitschinger [ 02/Jun/14 ]
merged into master




[JCBC-403] Fix undetected streaming disconnect on pure-view workloads Created: 22/Jan/14  Updated: 06/Feb/14  Resolved: 06/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.3.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-461] Use alternative constructor to allow configuration of max New I/O client Worker thread count Created: 21/May/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Vanessa Ablaya Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Linux, Java 1.6


 Description   
We are running multiple instances of an application on a blade with 24 cores. Netty creates (2 * #PROCESSORS) New I/O Client Worker threads per couchbase client. This is causing high thread count, as each of our application instances does not have complete access to the resources 24 processors. I want to be able to configure the maximum number of threads Netty creates.

Line 100 of BucketMonitor.java uses the following constructor from the netty library:

 public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor)


I would like it to use the alternative constructor that netty provides, so that I can configure the workerCount value:

public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor, int workerCount)



 Comments   
Comment by Michael Nitschinger [ 22/May/14 ]
That makes total sense. I'd rather not add an additional config setting for 1.* since the whole thing is revamaped for the 2.0 where the complete IO layer is based on netty and fully tunable.

What I could imagine though is a system property that you can set, which we could document. So people which run on those very powerful boxes can set it, and the others just don't have to care.
What do you think?
Comment by Michael Nitschinger [ 02/Jun/14 ]
Actually, I think we can trim that down since we are using it for streaming connections only. No need to provide that much. I'll investigate further and see what I can get into 1.4.2.
Comment by Michael Nitschinger [ 02/Jun/14 ]
Btw, two more things:

1) If you go to 1.4.1 and higher, it uses a different config fetching mechanism, so you should not see any of those threads created if you go with a server 2.5.0 and higher.
2) Even on your version, netty should create those threads only when needed, so do you mean you create lots of CouchbaseClient instances? You should reuse them as a singleton. Or do you see lots of those created even for one CouchbaseClient object? If so, this is a bug I think.

Can you provide more details on that? Maybe with a thread dump?
Comment by Michael Nitschinger [ 02/Jun/14 ]
http://review.couchbase.org/#/c/37722/
Comment by Michael Nitschinger [ 02/Jun/14 ]
We know only use one, so no need to configure. Please create a new ticket if you still then see issues with massive amount of threads generated, this is something else then for sure.




[JCBC-460] New configuration gets falsely discarded if only replica vbuckets change. Created: 12/May/14  Updated: 16/May/14  Resolved: 16/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-459] Ignore Configurations with a smaller or equal revid (if set) Created: 12/May/14  Updated: 16/May/14  Resolved: 16/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0, 1.4.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-446] After all nodes have been tried on NMVB, clean out the list to start over again. Created: 04/Apr/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-439] Fix overriding of the auth descriptor in the builder Created: 26/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-436] Expose custom auth timeout setting Created: 24/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-431] Add tainted config poller for carrier configs Created: 18/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.0-dp1
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-427] Unintended rollback to old configuration on http disconnect. Created: 11/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0-dp1
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-426] add envvar or property to force use of HTTP or Carrier Publication style bootstrapping Created: 10/Mar/14  Updated: 15/Apr/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0-dp1
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
For testing as well as in-the-field workarounds for particular cases, please add a feature to be able to force one type of configuration or another.




[JCBC-389] Make PersistTo/ReplicateTo really async and not block Created: 16/Dec/13  Updated: 16/Dec/13  Resolved: 16/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.3.0
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates JCBC-361 Add capability of async+durability wi... Resolved
is duplicated by JCBC-361 Add capability of async+durability wi... Resolved

 Comments   
Comment by Michael Nitschinger [ 16/Dec/13 ]
See JCBC-361




[JCBC-380] alternate bootstraps have a hardcoded port 8091 Created: 15/Nov/13  Updated: 28/Nov/13  Resolved: 28/Nov/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.9, 1.2, 1.2.1, 1.2.2
Fix Version/s: 1.2.3
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Since it is possible, though not easy, to make Couchbase run on a port other than 8091, the hostnames extracted from the configuration can't necessarily be on port 8091. The initial bootstrap port number should be applied to all of these derived URIs.

 Comments   
Comment by Michael Nitschinger [ 16/Nov/13 ]
Two things here

- IMHO this is an enhancement, since its a new requirement to run against != 8091
- Not sure if we can grab it from the initial list because things can change, picking it up from the new config may be better - hope the server supplies it correctly.
Comment by Michael Nitschinger [ 16/Nov/13 ]
Is there a way to change the ports to test it?
Comment by Michael Nitschinger [ 18/Nov/13 ]
http://review.couchbase.org/#/c/30372/




[JCBC-373] ReplicaGetFuture is not thread safe. Created: 28/Oct/13  Updated: 31/Oct/13  Resolved: 31/Oct/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.1.8, 1.1.9, 1.2, 1.2.1
Fix Version/s: 1.2.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-372] ReplicaRead can cause Huge Heap/StackOverflowException. Created: 28/Oct/13  Updated: 19/Dec/13  Resolved: 19/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Under specific scenarios, ReplicaReads can cause "inifinte" redispatch and therefore either overflow on the stack or cause huge heaps (created by large callbacks as a result).

 Comments   
Comment by Michael Nitschinger [ 19/Dec/13 ]
This has been fixed in the 12.3 and 1.2.2 series.




[JCBC-369] observePoll has wrong logic on delete Created: 15/Oct/13  Updated: 31/Oct/13  Resolved: 31/Oct/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.2
Fix Version/s: 1.2.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-368] Deadlock in BucketMonitor.startMonitor() on CountDownLatch when channel creation fails. Created: 14/Oct/13  Updated: 24/Oct/13  Resolved: 24/Oct/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.9, 1.2
Fix Version/s: 1.2.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deniss Afonin Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by JCBC-326 Couchbase client stuck indefinitely i... Resolved

 Description   
Deadlock in BucketMonitor.startMonitor() on CountDownLatch when channel creation fails.

Before deadlocking it also logs following exception (which by itself is another bug):

java.lang.IllegalStateException: An Executor cannot be shut down from the thread acquired from itself. Please make sure you are not calling releaseExternalResources() from an I/O worker thread.
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:73)
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:49)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.releaseExternalResources(NioClientSocketChannelFactory.java:180)
at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:319)
at com.couchbase.client.vbucket.BucketMonitor$1.operationComplete(BucketMonitor.java:193)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:428)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:419)
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:381)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processConnectTimeout(NioClientSocketPipelineSink.java:394)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:289)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)



 Comments   
Comment by Michael Nitschinger [ 14/Oct/13 ]
Hi, Thanks for reporting.

can you give me more details about your env? Particulary, which sdk version do you run and on which platform/os?
Comment by Deniss Afonin [ 14/Oct/13 ]
Yeah, sorry for missing that. Here it is:

couchbase-client-1.2.0

running with

java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

on OS X 10.8.5
Comment by Michael Nitschinger [ 14/Oct/13 ]
I see what's going on there, but I'd like to add a test to reproduce. How are you running into this?
Comment by Deniss Afonin [ 14/Oct/13 ]
I can't always reproduce that, this happens from time to time.
Comment by Michael Nitschinger [ 14/Oct/13 ]
I see, thanks for reporting it. Look forward to a proper fix in the next bugfix release.
Comment by Michael Nitschinger [ 14/Oct/13 ]
I think I fixed it, but in any case something went wrong during connection to the streaming connection.

Does the error is expected to happen in your environment or was it considered stable? I wonder in which circumstances you saw that. Even with my fix, its just shutting down properly but still the streaming conn attachment won't succeed.
Comment by Deniss Afonin [ 14/Oct/13 ]
Well, this happens for us only when couchbase server is not in local network i.e. we are connecting via internet with ~100ms latency and that internet connection might not be that stable.

Issue JCBC-326 might also be related to this.
Comment by Michael Nitschinger [ 14/Oct/13 ]
Yes that's what I assumed. So the thing is, in general we don't recommend running your app servers and db servers across the internet (they should be in the same datacenter, especially if you care about latency).

But this shouldn't be a limitation of the client itself. So I have a bugfix for this upcoming, but I'll also see what I can do to add some more retry logic to try different nodes from the list so if one fails we eventually connect to another one.
Comment by Michael Nitschinger [ 24/Oct/13 ]
fix has been merged into master and will be available in 1.2.2.




[JCBC-366] Properly allow override of metrics and listeners Created: 07/Oct/13  Updated: 11/Oct/13  Resolved: 11/Oct/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.2
Fix Version/s: 1.2.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-350] Race Condition in observe/replica Created: 27/Aug/13  Updated: 11/Sep/13  Resolved: 11/Sep/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.9
Fix Version/s: 1.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Michael Nitschinger [ 27/Aug/13 ]
Pushed into master, awating reporter feedback for close.




[JCBC-351] Enhance ReplicaRead with active node Created: 27/Aug/13  Updated: 13/Sep/13  Resolved: 13/Sep/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.9
Fix Version/s: 1.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-337] add new method for retrieving configuration over memcached binary protocol Created: 30/Jul/13  Updated: 07/Apr/14  Resolved: 22/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.0-dp1
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: cccp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
depends on MB-8728 add a new operation which allows for ... Closed

 Description   
To support better faster, more reliable response to configuration changes, the Java client should change from HTTP streaming configuration to the new memcached binary protocol method delivered under project CCCP.

In Java, this will necessitate a new constructor to start from hostname with an optional port number, as all current constructors are intended to be RESTful and use URIs.

This is part of project CCCP, as covered at http://www.couchbase.com/wiki/display/couchbase/Cluster+Configuration+Carrier+Publication

 Comments   
Comment by Michael Nitschinger [ 22/Feb/14 ]
I'm closing this since it has been merged technically. Let's open bug reports for all specific issues we identify.




[JCBC-336] IndexOutOfBounds when config changes during observePoll Created: 30/Jul/13  Updated: 01/Aug/13  Resolved: 01/Aug/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.7
Fix Version/s: 1.1.9
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck:604
at java.util.ArrayList.get:382
at com.couchbase.client.vbucket.config.DefaultConfig.getServer:81
at com.couchbase.client.vbucket.VBucketNodeLocator.getServerByIndex:113
at com.couchbase.client.CouchbaseClient.observePoll:2185
at com.couchbase.client.CouchbaseClient.set:1286




[JCBC-324] Avoid NPE when Host is Up, but no CB is running Created: 02/Jul/13  Updated: 16/Dec/13  Resolved: 16/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by JCBC-321 Null Pointer in client when deleting ... Closed
is duplicated by JCBC-379 If bucket details are incorrect, erro... Closed

 Description   
When the target host is reachable, but no server is running, the client throws a unhelpful NPE:

Caused by: java.lang.NullPointerException
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:148)
at com.couchbase.client.CouchbaseConnectionFactory.getVBucketConfig(CouchbaseConnectionFactory.java:231)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:237)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:175)

The logs also show more infos, but this NPE is not helpful in the stack.

 Comments   
Comment by Brad Wood [ 02/Jul/13 ]
Additional info here:
http://www.couchbase.com/communities/q-and-a/sdk-throws-unhelpful-npe-when-server-unreachable
Comment by Michael Nitschinger [ 18/Nov/13 ]
Brad, are you sure this is not fixed with recent releases?

I tried
 - not running node
 - running node with wrong bucket name
 - running node with wrong password

and I always get com.couchbase.client.vbucket.ConfigurationExceptions and not NPEs anymore... this is with vanilla 1.2.2 but I remeber fixing it earlier.

If you still see it, can you get me more information on the environment?




[JCBC-319] Force Resubscribing in CouchbaseMemcachedConnection Created: 21/Jun/13  Updated: 19/Jul/13  Resolved: 02/Jul/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.8
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
blocks JCBC-305 Client does not recover with Memcache... Closed




[JCBC-312] Reconfigure not triggered on Master -1 Created: 31/May/13  Updated: 07/Jun/13  Resolved: 07/Jun/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.1.7
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
During some failover/rebalance scenarios, it could be the case that no master is responsible for the document. While this should not be the case, it is observed in scenarios where the client may still have an outdated config from somewhere.

This leads to RuntimExceptions raised, but reconfigure is never actively triggered. In QE tests, this manifests itself in errors during change and rebound.

While it should be elsewhere investigated how those -1 get in place, checking for this and triggering reconfigure is a safety net for running operations.

 Comments   
Comment by Michael Nitschinger [ 31/May/13 ]
No fix applied:

Will show phase timings..
--------------------------
Phase statistics for RAMP
  OK/sec: 3619

  OK: 108596
  ERR: 0
Phase statistics for CHANGE
  OK/sec: 3204

  OK: 999917
  ERR: 141386
Phase statistics for REBOUND
  OK/sec: 3966

  OK: 357026
  ERR: 26898
---------------------

Fix applied:

--------------------------
Phase statistics for RAMP
  OK/sec: 3536

  OK: 106102
  ERR: 0
Phase statistics for CHANGE
  OK/sec: 1870

  OK: 471474
  ERR: 825
Phase statistics for REBOUND
  OK/sec: 4063

  OK: 365714
  ERR: 0
---------------------

stester run: /stester -C 127.0.0.1:8050 -i 20devcluster.ini -c failover.Once --vdsw_dvname ddoc/vquery --hdsw_http_threads 5 --grace_after 30 --ept 1 --ramp 30 --num_nodes 2 --hdsw_mc_threads 10 --workload dsw.Hybrid --action_delay 10 --hdsw_cb_threads 10 --action FO_REBALANCE --dsw_timeres 1 -d -o viewlog_3_f.out
Comment by Michael Nitschinger [ 31/May/13 ]
http://review.couchbase.org/#/c/26636/
Comment by Michael Nitschinger [ 31/May/13 ]
Note that before this change, the RuntimeException bubbled up to the userlevel, blocked everything there - but more importantly, cf.checkConfigUpdate(); never got triggered!
Comment by Michael Nitschinger [ 31/May/13 ]
This also improves this scenario run:

Effective stester command line
    -C 127.0.0.1:8050 \
    -i 20devcluster.ini \
    -c failover.Once \
    --vdsw_dvname ddoc/vquery \
    --hdsw_http_threads 5 \
    --grace_after 30 \
    --ept 1 \
    --ramp 30 \
    --num_nodes 2 \
    --hdsw_mc_threads 10 \
    --workload dsw.Hybrid \
    --action_delay 10 \
    --hdsw_cb_threads 10 \
    --action FO_REBALANCE \
    --dsw_timeres 1 \
    -d \

--------------------------
Phase statistics for RAMP
  OK/sec: 3057

  OK: 91713
  ERR: 0
Phase statistics for CHANGE
  OK/sec: 3172

  OK: 957997
  ERR: 187000
Phase statistics for REBOUND
  OK/sec: 4108

  OK: 369758
  ERR: 63598
---------------------

After

Will show phase timings..
--------------------------
Phase statistics for RAMP
  OK/sec: 3453

  OK: 103594
  ERR: 0
Phase statistics for CHANGE
  OK/sec: 2346

  OK: 731968
  ERR: 549
Phase statistics for REBOUND
  OK/sec: 4064

  OK: 365817
  ERR: 0
---------------------




[JCBC-280] Provision for auth failure in case of calling createBucket in the same txn twice and adding updateBucket functionality Created: 04/Apr/13  Updated: 12/Jun/13  Resolved: 03/Jun/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deepti Dawar Assignee: Deepti Dawar
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There are three ways using which we are creating connection in the java client to the server.

1) ClusterManager
2) BucketTool

Both of these classes internally call the ClusterManager.createBucket for creation of the bucket.

Now if I am using all the three instances of the above classes in a single function, the bucket information is overridden without any checks. Ideally using any of the connection classes if I have created a SASL bucket with some information like bucket name = 'SaslBucket', bucket password = 'password', I should not be allowed to change the password using instance of another class. There should be auth failure error being returned the second time client tries to connect to the same bucket. Server supports this because there is an Edit Bucket functionality at the server for the same.

There should be a means of distinction in the request that we want to create the bucket or update.
In case of bucket creation duplicity should be checked where as in case of update this should be allowed as is.

Also, the expectation is that, the user might update the bucket information if he requires to change the password or other details, by explicitly calling updateBucket method which is currently not available.

 Comments   
Comment by Deepti Dawar [ 05/Apr/13 ]
Alright.

Consider the following piece of code :

public void testCreateSaslBktBadPswd() throws Exception {
manager.createNamedBucket(BucketType.COUCHBASE, "bucket1", 100, 0,
"password", false);
List<URI> uris = new LinkedList<URI>();
uris.add(URI.create("http://"
+ TestConfig.IPV4_ADDR + ":8091/pools"));
new CouchbaseConnectionFactory(uris, "bucket1", "password12");
new CouchbaseClient(uris, "bucket1", "password123");
BucketTool bt = new BucketTool();
bt.createSaslBucket("bucket1", BucketType.COUCHBASE, 1, 2, true);
  }

Here, using cluster manager instance, we create a SASL bucket i.e. bucket1 with password as password.
Next you would see the usage of CouchbaseConnectionFactory instance which is trying to manipulate the same bucket, bucket1. It overrides the password to password12. Here it internally calls createBucket again without checking whether such a bucket already exists or not.
Another attempt is made to connect using the CouchbaseClient instance. The bucket password is again changed.
Similarly, the bucket tool instance also does the same.

Instead, there should be auth failure error being returned the second time client tries to connect to the same bucket.
Also, the expectation is that, the user might update the bucket information if he requires to change the password or other details, by explicitly calling updateBucket method which is currently not available.
Comment by Michael Nitschinger [ 24/Apr/13 ]
Is this still an issue?
Comment by Deepti Dawar [ 25/Apr/13 ]
You mean there has been a fix entered for this ?
Comment by Michael Nitschinger [ 25/Apr/13 ]
No, because I think the ticket is somewhat misleading. The Factory and the Client have nothing to do with Bucket creation.

The Manager should be used for clients and the bucket tool is an internal tool used in our testing environment. I still don't get whats the problem here? Just pick one of these depending on the thing you want to do.
Comment by Deepti Dawar [ 25/Apr/13 ]
The problem here is that, lets say there are two users - User 1 and User 2.
User 1 created the instance using CouchbaseConnectionFactory and provided a password to the default bucket as 'password'
User 2 created the instance using CouchbaseClient and provided a password as 'password2'

As both internally go and create the bucket, don't you think the two users are overriding each other's bucket information ?
Shouldn't the second user be returned an auth failure for the same ?
Comment by Michael Nitschinger [ 25/Apr/13 ]
Hi Deepti,

I'm not sure I can follow you. Neither CouchbaseConnectionFactory nor the CouchbaseClient class creates a bucket by any means. The user has to explicitly use the Manager to create a bucket! And the BucketTool uses the Manager underneath as well, it just provides some convenience methods.
Comment by Deepti Dawar [ 25/Apr/13 ]
Ok.

I see that the conflict is between -

manager.createNamedBucket
bucketTool.createSaslBucket

both of which call the same createBucket method.
Comment by Michael Nitschinger [ 25/Apr/13 ]
Ok so now that we pinned that down.

Can you please

1) clarify whats the issue between those methods and what needs to be fixed
2) update the title and description of the ticket to reflect those changes?

Thanks!
Comment by Deepti Dawar [ 28/May/13 ]
http://review.couchbase.org/#/c/26557/
Comment by Matt Ingenthron [ 11/Jun/13 ]
Deepti: any updates on the underlying issue? Thanks.
Comment by Deepti Dawar [ 12/Jun/13 ]
Hi Matt,

I have already checked in the code and it is up for review in gerrit.

Here is the link - http://review.couchbase.org/#/c/26557/

Regards,
Deepti
Comment by Matt Ingenthron [ 12/Jun/13 ]
Yes, but there's already a review on the patch set indicating there's a problem. Please update the patchset, clarify why what is being observed is expected or get with the reviewer to understand the problem if it's not clear.

Thanks!
Comment by Deepti Dawar [ 12/Jun/13 ]
Ah Yes ! Will update in some time.




[JCBC-276] Client does not detect silently dying Streaming Node Created: 20/Mar/13  Updated: 27/May/13  Resolved: 27/May/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4
Fix Version/s: 1.1.7
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When connected to the EPT/streaming node and the node is "frozen" or dies silently otherwise (doesn't force the closing of the chunked socket), the connection stays established.

This can easily be reproduced outside of the client by connecting the browser to the streaming URL and then freezing a VM. The browser will still "spin" and wait for new chunks to come up.

The proposed solution is to have a netty handler in place that raises a exception when there is not traffic for N number of seconds (like 30) over the streaming connection. After this is detected, we have two possibilities:

- reconnect completely, but this involves lots of overhead every 30 seconds.
- send a HTTP HEAD packet and only if this doesnt work out reconnect. This means in the normal case we only have a HTTP HEAD request sent every 30 seconds, not much overhead.
If this fails, we then trigger the reconfigure.

Netty has a ReadTimeoutHandler to help with this. My POC already kinda works, I just need to find a way to properly distinguish the HEAD response on the ResponseHandler from regular chunks that arrive from the same channel.

 Comments   
Comment by Matt Ingenthron [ 20/Mar/13 ]
I had fixed this with couchbase buckets and we test for this in SDKQE. Is this possibly isolated to memcached buckets.
Comment by Michael Nitschinger [ 21/Mar/13 ]
Actually, this has been discovered while using Couchbase buckets. Let's chat about this, but I think thats a different issue and not related to it.
Comment by Matt Ingenthron [ 21/Mar/13 ]
Sure, the solution previously was to have that threshold if we were getting unexpected failures. Once we pass that threshold, we'd try to re-subscribe.

I'm not opposed to a heartbeat, but I don't think the HTTP HEAD is good, since the mochiweb erlang implementation is effectively the same as a GET. I'll reach you and chat through it.
Comment by Michael Nitschinger [ 21/Mar/13 ]
Okay after talking this through with Matt, I reran the script to see if the proposed solution (increasing ops/s) fixed the problem.

Interestingly, it turns out when doing set/get's it never hits our anticipated codepath but instead the get operations just time out, the threshold never gets increased and nothing happens. This is something we need to reinvestigate.
Comment by Michael Nitschinger [ 27/May/13 ]
Closing this for now, because our fallback algorithm works as expected. The threshold bug never triggered has been resolved in a different ticket.




[JCBC-274] Jenkins test results failing. Created: 16/Mar/13  Updated: 19/Jul/13  Resolved: 19/Jul/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File git_build_description.png     PNG File issues_latest_run_jenkins.png     PNG File latest_build.png    
Issue Links:
Dependency

 Description   
When the jenkins job for java build is run with the latest source in Git and the tests are run, 6 tests are failing. Those are very obvious errors and can be fixed. Screenshot attached.

 Comments   
Comment by Deepti Dawar [ 18/Mar/13 ]
Michael, kindly check why the NPEs are coming in these results.

Thanks
Deepti.
Comment by Deepti Dawar [ 04/Apr/13 ]
Please review few fixes at :

http://review.couchbase.org/#/c/25482/
Comment by Michael Nitschinger [ 19/Jul/13 ]
This has been solved in general, if there are specific other tests failing lets reopen specific issues for them.




[JCBC-272] removing non-bootstrap node with memcached bucket okay, but adding the same node results in failures Created: 13/Mar/13  Updated: 15/Mar/13  Resolved: 15/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.1.5
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Client 1.1.2 with required spymemcached 2.8.11. Server 2.0.0 with a single authenticated memcached bucket. Assertions enabled.

Issue Links:
Duplicate
duplicates JCBC-271 Adding a node to an existing cluster ... Resolved

 Description   
From the attached log, one will see first the removal of a node (click remove -> click rebalance), and things are generally okay.

Following that, you'll see the node added (via the config wizard, joining to node 192.168.1.200), and one configuration received, but nothing actually used yet.

Following that, you'll see the node rebalanced in with a set of assertion errors and then workload ceases.

Notably:
2013-03-13 19:29:03.924 INFO com.couchbase.client.CouchbaseMemcachedConnection: Reconnecting {QA sa=/192.168.1.201:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 139891 Key: pool-25-thread-1:48454, topWop=null, toWrite=0, interested=1}
Exception in thread "Memcached IO over {MemcachedConnection to /192.168.1.200:11210 /192.168.1.202:11210 /192.168.1.201:11210}" java.lang.AssertionError: Attempting to overwrite channel

 Comments   
Comment by Deepti Dawar [ 14/Mar/13 ]
In this scenario, I am encountering something like this :

[INFO 33.11 cbsdk.scenario failover.py:198] Sleeping for 60 seconds after failover
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.MemcachedConnection handleIO
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to exception on {QA sa=/10.3.2.57:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149, topWop=null, toWrite=0, interested=1}
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] java.io.IOException: Disconnected unexpected, will reconnect.
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:526)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:454)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158)
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.MemcachedConnection queueReconnect
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.2.57:11210, #Rops=1, #Wops=0, #iq=1, topRop=Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149, topWop=null, toWrite=0, interested=1}, attempt 0.
[SDKD(WARNING) 33.12 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 5552 Key: SimpleKey__REP_149
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] java.lang.RuntimeException: Cancelled
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:169)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:132)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:64)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
[SDKD(WARNING) 33.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
[SDKD(WARNING) 33.16 cbsdk.sdkd.local executor.py:66] Mar 14, 2013 2:17:37 AM com.couchbase.client.CouchbaseMemcachedConnection reconfigure
[SDKD(WARNING) 33.17 cbsdk.sdkd.local executor.py:66] INFO: Scheduling Node /10.3.2.57:11210for shutdown.




[JCBC-270] client does not handle failure of EPT node with memcached bucket Created: 13/Mar/13  Updated: 27/May/13  Resolved: 27/May/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.2
Fix Version/s: 1.1.7
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3 node cluster of 2.0, one memcached bucket authenticated, Couchbase Java Client 1.1.2 with required spymemcached 2.8.11

Attachments: Text File CBSE-472repro-2.txt    

 Description   
When investigating a case, it seems that the dropped configuration when using a memcached bucket is never reestablished. I'm not sure if this is because we rely on timeouts (which we won't get in this case) or something else.

Steps to reproduce:
1. Set up three node cluster
2. Run a constant workload, starting off of one of the nodes (192.168.1.200 in my config)
3. Remove that node from the cluster

Observed behavior:
The client sees the config dropped, but doesn't reconfigure.

Expected behavior:
Client bootstraps off of one of the other nodes.

Attached file shows the config log in this case.

 Comments   
Comment by Michael Nitschinger [ 14/Mar/13 ]
Hey Matt,

one quick update.. I don't know yet if its related, but you've been using the wrong netty version.. your logs show /Users/ingenthr/lib/netty-3.2.5.Final.jar' but the correct one is 3.5.5!

Since netty handles the streaming connection, this may be related - I dont know yet.
Comment by Michael Nitschinger [ 15/Mar/13 ]
Matt, with the change proposed in JCBC-271 and using Netty 3.5.5, I dont see this behaviour (anymore). I tried with 3.2.5 but my connections always go from unbound to connected after some time, even when I wildly failover/rebalance - as it should be. It even waits as we implemented it when I remove both EPT nodes for some time and then add them back it comes back nicely.

Can you try to repro with those two changes and if it still fails lets do a quick screen sharing.
Comment by Matt Ingenthron [ 18/Mar/13 ]
Indeed, I could not reproduce this one after moving to the right dependencies.




[JCBC-271] Adding a node to an existing cluster causes issues with a running memcached bucket Created: 13/Mar/13  Updated: 16/Mar/13  Resolved: 16/Mar/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.2
Fix Version/s: 1.1.5
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3 node cluster of 2.0, one memcached bucket authenticated, Couchbase Java Client 1.1.2 with required spymemcached 2.8.11

Attachments: Text File CBSE-472repro.txt    
Issue Links:
Duplicate
duplicates JCBC-267 Memcached bucket still fails with all... Closed
is duplicated by JCBC-272 removing non-bootstrap node with memc... Resolved

 Description   
When adding a node to a cluster, I'd observed a number of AssertionErrors and some very odd errors about connections being null.

This popped up when I wasn't expecting an issue, so I'm a bit surprised. The scenario I think is:

Steps to reproduce:
1. Have two nodes running (.200 and .201 in my case here)
2. Add another node (.202 in my case here)

See the logs for what detail is available.

 Comments   
Comment by Michael Nitschinger [ 15/Mar/13 ]
This is the same root cause, the channel gets overriden on add.
Comment by Michael Nitschinger [ 15/Mar/13 ]
http://review.couchbase.com/#/c/25171/
Comment by Matt Ingenthron [ 16/Mar/13 ]
Yes, that fixed the issue. I had some issues on adding that node back, but there was a problem at the cluster side.




[JCBC-266] NPE in ConfigurationProviderHTTP needs to be handled. Created: 12/Mar/13  Updated: 02/Jul/13  Resolved: 02/Jul/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.1.8
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
NPE in ConfigurationProviderHTTP needs to be handled.
This is seen at quite a lot of places.

Please refer to the integration test results at :

https://docs.google.com/a/globallogic.com/spreadsheet/ccc?key=0AmLvaJ8oRZ-TdHJBNHJ0eGZfbjBaenBSVGM2VS1oNlE#gid=0


Stack Trace :

SDKD: Mar 12, 2013 1:55:39 AM com.couchbase.client.CouchbaseConnectionFactory$Resubscriber run
SDKD: WARNING: Resubscribe attempt failed:
SDKD: java.lang.NullPointerException
SDKD: at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:149)
SDKD: at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:319)
SDKD: at com.couchbase.client.CouchbaseConnectionFactory$Resubscriber.run(CouchbaseConnectionFactory.java:405)
SDKD: at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
SDKD: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
SDKD: at java.lang.Thread.run(Thread.java:619)
SDKD: Mar 12, 2013 1:55:39 AM com.couchbase.client.CouchbaseConnectionFactory$Resubscriber run

 Comments   
Comment by Michael Nitschinger [ 12/Jun/13 ]
http://review.couchbase.org/#/c/26862/




[JCBC-231] Handle View Errors (Vbucket & Merge) appropriately Created: 02/Feb/13  Updated: 19/Jul/13  Resolved: 19/Jul/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0, 1.1.1
Fix Version/s: 1.1.8
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates JCBC-313 View Requests >= HTTP 300 should be r... Resolved
is duplicated by JCBC-307 Hybrid workload with rebalance scenar... Resolved

 Description   
During rebalance out it can happen that the view node is still able to service http connections, but fails with the following errors:

- error Reason: A view spec can not consist of merges exclusively.
or
- no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets

If this is happening, the client needs to deal with it. Appropriate solution needs still to be dicussed, but one approach could be to retry on a different node. Note that this needs to work for both async and sync calls.

 Comments   
Comment by Michael Nitschinger [ 02/Feb/13 ]
The corresponding server ticket can be found here: http://www.couchbase.com/issues/browse/MB-7661
Comment by Michael Nitschinger [ 19/Jul/13 ]
See JCBC-313 which now retries these kind of responses properly.




[JCBC-230] View Connection only released when successful (not on failure) Created: 01/Feb/13  Updated: 06/Feb/13  Resolved: 06/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0, 1.1.1
Fix Version/s: 1.1.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When a view operation is cancelled (timeout or else), the underlying connection for the ConnectionManager is not released and as a result no new view request can go over it.

 Comments   
Comment by Michael Nitschinger [ 06/Feb/13 ]
https://github.com/couchbase/couchbase-java-client/commit/4b733eb37351aa62bab7c6d58d7f689f829ea704




[JCBC-227] client behaves poorly if entire URI list is not available Created: 31/Jan/13  Updated: 01/Feb/13  Resolved: 01/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.1
Fix Version/s: 1.1.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Currently, if the client has no valid URIs in it's list, it's reported that it will spin trying to connect nonstop. We should add a backoff if the entire list is traversed and unavailable. An exponential backoff (do-while style) with a ceiling and warning to a log is probably best.

 Comments   
Comment by Michael Nitschinger [ 31/Jan/13 ]
http://review.couchbase.org/#/c/24322/




[JCBC-198] Using ReplicateTo.ONE after node-failover leads to index out of bounds Created: 28/Dec/12  Updated: 27/Feb/13  Resolved: 27/Feb/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0
Fix Version/s: 1.1.3
Security Level: Public

Type: Bug Priority: Critical
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
using couchbase client library version 1.1.0, server version 2.0.0 community edition (build-1976-rel).
The cluster configuration remains same as in my initial email - three servers with "Enable Failover" property ON, a dedicated port "couchbase" bucket with 1 replica enabled.

When stopping a server the client fails to communicate with the cluster with several errors.

Client initializing -

List<URI> serverList = parseConnectionProperties(port, hosts);
CouchbaseConnectionFactory cf = new CouchbaseConnectionFactory(serverList, cacheName, "");
client = new CouchbaseClient(cf);

Usage of the client to write data -

public void put(final String key, final String value) {
     client.set(key, expirationSeconds, value, PersistTo.MASTER, ReplicateTo.ONE);
 }

During the tests, when cluster is not available, I did a thread dump to the application , see the print below (ajp--127.0.0.1-8020-1@8796). Is it possible that node that is down is in some way a "master" of the data, and since the client.set() method uses PersistTo.MASTER parameter the things do not work?

"ajp--127.0.0.1-8020-1@8796" daemon prio=5 tid=0x8c nid=NA waiting
  java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1650)
at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1750)
at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1199)
at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
at java.lang.Thread.run(Thread.java:722)

The errors i get during the test after a server was failed over -

 
17:45:41,911 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-4] Verify Sso Login failed. sessionId - n2b+0EZEzTaM7a64gKKS7k+L.undefined [com.liveperson.liveEngage.ssoIdpLogin.LoginVerification]
17:45:41,911 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.RuntimeException: Timed out waiting for operation
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:134)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1188)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
17:45:41,917 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)
17:45:41,917 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: tlv-le-couchbase-int1.tlv.lpnet.com/192.168.24.184:11210
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:159)
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:132)
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) ... 28 more

OR

19:21:47,997 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-3] Verify Sso Login failed. sessionId - 6tJElgOsTUcr-7lWgJlWh2HV.undefined [com.liveperson.liveEngage.ssoIdpLogin.LoginVerification]
19:21:47,997 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.ArrayIndexOutOfBoundsException: -1
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.elementData(ArrayList.java:371)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.get(ArrayList.java:384)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.config.DefaultConfig.getServer(DefaultConfig.java:81)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.VBucketNodeLocator.getServerByIndex(VBucketNodeLocator.java:112)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1621)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1750)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1199)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)

Hopefully it's the same one, but just in case, here's another one:
17:41:34,231 ERROR [stderr] (Couchbase View Thread for node tlv-le-couchbase-int2.tlv.lpnet.com/192.168.24.185:8092) 2012-12-10 17:41:34.231 INFO com.couchbase.client.ViewNode: I/O reactor terminated for tlv-le-couchbase-int2.tlv.lpnet.com
17:42:23,120 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-2] Exception while trying to remove data from cache for session id: y6mpopojZkaJDcWeRDL4J0xp.undefined [org.apache.jsp.views.logout_jsp]
17:42:23,121 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
17:42:23,121 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.rangeCheck(ArrayList.java:604)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.get(ArrayList.java:382)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.config.DefaultConfig.getServer(DefaultConfig.java:81)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.VBucketNodeLocator.getServerByIndex(VBucketNodeLocator.java:112)
17:42:23,123 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1624)
17:42:23,123 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1753)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.delete(CouchbaseClient.java:1113)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.remove(CouchbaseReadWriteCache.java:28)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.remove(CouchbaseReadWriteCache.java:17)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.removeDataFromCache(LEUsersCacheManager.java:132)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jsp.views.logout_jsp._jspService(logout_jsp.java:82)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:326)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:253)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.ExceptionFilter.doFilter(ExceptionFilter.java:26)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,129 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,129 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.SSOAuthorizationFilterChain.doFilter(SSOAuthorizationFilterChain.java:30)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:65)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,131 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,131 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
17:42:23,132 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,132 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
17:42:23,135 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
17:42:23,138 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
17:42:23,140 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)




[JCBC-190] Allow ComplexKeys to work with Floats (not only Doubles) Created: 19/Dec/12  Updated: 18/Jan/13  Resolved: 18/Jan/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1.0
Fix Version/s: 1.1.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
See: http://www.couchbase.com/forums/thread/set-floating-point-key-couchbase-client-1-1-0-jar

 Comments   
Comment by Michael Nitschinger [ 28/Dec/12 ]
http://review.couchbase.com/#/c/23605/




[JCBC-165] ComplexKey does not support partial compound keys with single field Created: 04/Dec/12  Updated: 06/Dec/12  Resolved: 06/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.1-beta
Fix Version/s: 1.1.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Chris Tashjian Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I can't figure out how to do partial compound keys with a single value --

ex. 1 - Specifying both parts of the key:
http://localhost:8092/bucketname/_design/mydesigndoc/_view/myview?group_level=1&startkey=["0",1351742400000]&endkey=["Z",1353073898844]

ex. 2 - Only specifying the first part of the key:
http://localhost:8092/bucketname/_design/mydesigndoc/_view/myview?group_level=1&startkey=["0"]&endkey=["Z"]

I can get ex. 1 to work via the Java client library, but I've had no luck with ex. 2. I've tried setting ranges, complex keys and regular keys. The closest I get is a query string where the "[" and "]" have been escaped which results in a bad URL. I was only able to get this far by manually concatenating the "[" and "]" onto my key and using the query.setKey(String) method.

I believe that ComplexKey should be able to handle ex 2 by calling ComplexKey.of("0") and ComplexKey.of("Z").

 Comments   
Comment by Michael Nitschinger [ 04/Dec/12 ]
Correct, I came across it today as well.
Comment by Michael Nitschinger [ 05/Dec/12 ]
http://review.couchbase.org/#/c/23086/
Comment by Michael Nitschinger [ 06/Dec/12 ]
you can now use the forceArray method on the ComplexKey to do this! .. will be available in 1.1.0.




[JCBC-155] add javadoc for AbstractView, View, SpatialView Created: 27/Nov/12  Updated: 03/Dec/12  Resolved: 29/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When fixing up a test after a merge commit, I noted that I had to create a View object directly. In the process, I found that many of these classes are public, have public ctors, but have no javadoc. The client API takes AbstractView as an argument, so it may be tempting to just construct one. Is it correct to do so? Maybe. Javadoc doesn't tell me much.

The arguments to the ctors could also be expanded as they're a bit outside code style. Specifically, variable names should be meaningful and at least three or more characters if the variable is to be around for a while.

 Comments   
Comment by Michael Nitschinger [ 28/Nov/12 ]
http://review.couchbase.org/#/c/22872/
Comment by Michael Nitschinger [ 29/Nov/12 ]
Pushed to master, will be available in dp5/beta.




[JCBC-152] Document testing procedures for unit/functional tests Created: 21/Nov/12  Updated: 03/Dec/12  Resolved: 29/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Mark Nunberg Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
It would be good to have clear documentation in the repo about how to:

(1) run individual tests
(2) change cluster parameters
(3) write more tests

This might be trivial or obvious to do, but very much needed.

For an example of what should be available, see:

(This does not say how to run individual tests though)
https://github.com/couchbase/libcouchbase#run-the-testsuite-towards-a-running-cluster

A more extensive example for a more complex test suite:
https://github.com/couchbase/php-ext-couchbase/blob/master/TESTING.pod

I'm marking this as a 'library' bug because this would require some detailed knowledge of how the tests are written :)

 Comments   
Comment by Michael Nitschinger [ 28/Nov/12 ]
http://review.couchbase.org/#/c/22874/
Comment by Michael Nitschinger [ 29/Nov/12 ]
This has been fixed and is available under TESTING.md in the lib.




[JCBC-148] Issue with Observe API Persist.TWO and 1 dead node: Time Out when doing set operation Created: 18/Nov/12  Updated: 03/Dec/12  Resolved: 03/Dec/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Critical
Reporter: Tug Grall (Inactive) Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 2 nodes cluster on couchbase-server-community_x86_2.0.0-1947-rel
  1 node on Ubuntu (VM)
  1 node on OS X

Bucket configure with 1 replica

Attachments: Zip Archive CouchbaseSamples.zip    

 Description   
I have a very simple Java program that connect to the 2 nodes and do a set with the following code:

1. So I try to connect to multiple nodes
{code}
            List<URI> couchbaseServerUris = new ArrayList<URI>();
            couchbaseServerUris.add( new URI("http://192.168.0.108:8091/pools") );
            couchbaseServerUris.add( new URI("http://192.168.0.104:8091/pools") );
            CouchbaseClient client = new CouchbaseClient( couchbaseServerUris , "default" , "" );
{code}

2. Then I call the set operation
{code}

        OperationFuture<Boolean> stored = client.set( "my-dummy-key",0, "{\"name\" : \"foo\", \"title\" : \"bar-test\"}", PersistTo.TWO);
{code}

---
So everything is working as expected when the 2 nodes are up.

When I kill 1 node (for example : disconnecting, or stopping, or pausing the Ubuntu VM) I have the following behavior:


When I execute this program:
1- I have an exception saying that 1 node is down : Expected behavior (even if we could avoid a long stack trace)
{code}
2012-11-18 08:14:55.830 WARN com.couchbase.client.vbucket.ConfigurationProviderHTTP: Connection problems with URI http://192.168.0.108:8091/pools ...skipping
java.net.ConnectException: Host is down
{code}


2- When I do the set the program is stopped/blocked until it reaches a network timeout
2012-11-18 08:20:13.462 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
{code}
Error while storing : Observe Timeout - Polled Unsuccessfully for at least 40 seconds.
2012-11-18 08:20:13.466 INFO done : true
done : {OperationStatus success=false: Observe Timeout - Polled Unsuccessfully for at least 40 seconds.}
com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
2012-11-18 08:20:13.467 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
{code}


Note that it is only happening with PersistTo.TWO
if I use PersistTo.MASTER or PersistTo.ONE : the program is executed with no error and no stop
if I use PersistTo.THREE ( or more) : the program is executed, no stop with the expected observe message : ( Error while storing : Requested persistence to 3 node(s), but only 2 are available.
 )




 Comments   
Comment by Tug Grall (Inactive) [ 18/Nov/12 ]
Sample program
Comment by Matt Ingenthron [ 20/Nov/12 ]
I do believe that's actually expected behavior, but let's talk through it to get your opinion.

We have a couple of options in the state of unexpected failure. one is we try our hardest to get the operation requested of us done and we rely on timeouts to keep from blocking forever. The second is that we keep tabs on our connections, and if the connection is down, we fail operations immediately so as to not have the application code waiting for something that may or may not succeed.

Had you gone in and removed the second node (click 'remove' and 'rebalance'), then the client should have done something similar to when you requested three nodes. The failure you describe above is unexpected. Further, the client library doesn't really know if it's temporary or permanent.

Finally, I do want to note, and I think this is well documented, that many things with Observe protocol under them end in timeouts. This is not the only one. Generally speaking, application code should be ready to do *something* in the case of a timeout.
Comment by Matt Ingenthron [ 20/Nov/12 ]
Tug explained this further. The PersistTo.THREE check must be happening after doing some operations, which is a bit late considering this operation can never succeed. The failure should be the same with a cluster that has a down node as it is with a cluster that just doesn't have a primary and to replica locations.
Comment by Mike Wiederhold [ 21/Nov/12 ]
The way Rags wrote this code originally was to do the set and then the observe. The observe part is the part that does all of the checking so the set will actually go through an then you will get the error. Similarly there is no checking for downed nodes and I don't think we actually have the ability to do this at the moment, but I may be wrong.

On another note, one other thing I thing is wrong is returning an OperationFuture from all of the observe functions, but it isn't actually an asynchronous function.
Comment by Michael Nitschinger [ 30/Nov/12 ]
http://review.couchbase.org/#/c/22936/
Comment by Michael Nitschinger [ 03/Dec/12 ]
fixed and will be available in the beta release.




[JCBC-134] resubscriber IllegalArgumentException during topology changes Created: 19/Oct/12  Updated: 23/Jan/13  Resolved: 23/Jan/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.0, 1.1.1
Fix Version/s: 1.1.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Mark Nunberg Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Got this during a swap-rebalance. I don't have any other relevant information to reproduce for now (fwiw I've run this many times and it's the first time I'm seeing it.. though I've just seen it again on the very next run).

During the second run, the cluster rebalance is actually hanging..

Attachments: Text File daschl-4-rebealance_two_nodes.log     Text File daschl-4-restart.log     Zip Archive junit.zip     File log2.txt.bz2     File log2.txt.bz2    
Issue Links:
Dependency

 Description   
Exception in thread "couchbase cluster resubscriber - running" java.lang.IllegalArgumentException: Bucket name cannot be null and must never be re-set to a new object.
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:240)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.finishResubscribe(ConfigurationProviderHTTP.java:215)
        at com.couchbase.client.CouchbaseConnectionFactory$Resubscriber.run(CouchbaseConnectionFactory.java:322)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)


Unfortunately I don't have a whole lot more insight into what's happening, but the stack trace might be helpful to examine.. assigning to myself until I have more info..

 Comments   
Comment by Michael Nitschinger [ 08/Nov/12 ]
Hi Mark,

can you elaborate a bit more whats going on through the test? This particular exception can come up when the java sdk tries to subcribe to a new node when the old connection is closed. I think the scenario should give us a connection to what is happening at runtime in the java sdk.

Thanks,
Michael
Comment by Mark Nunberg [ 12/Dec/12 ]
I've run the Java SDKD tests several times already and cannot reproduce this. Will re-open it if i see it again
Comment by Mark Nunberg [ 21/Jan/13 ]
Seen again at:

http://review.couchbase.org/#/c/24092/
Comment by Mark Nunberg [ 21/Jan/13 ]
So as I mentioned in the bug, I closed it because I haven't seen this error. Just now, both me and Michael encountered this error while running the SDKD tests
Comment by Michael Nitschinger [ 22/Jan/13 ]
Attaching the logs for http://review.couchbase.org/#/c/24092 changeset 4 (prefixed with daschl-4-) on some test runs.

Deepti is currently running the changeset against the brun cluster, expect some info in 2 hours.
Comment by Michael Nitschinger [ 22/Jan/13 ]
./stester -i 20devcluster.ini --service ALL --svcaction RESTART --num_nodes 3 --no_fo 1 -c failover.Once --dsw_timeres 1 -d -o restart.log -C 127.0.0.1:8050
Comment by Michael Nitschinger [ 22/Jan/13 ]
./stester -i 20devcluster.ini -c rebalance.Once --mode out --rbcount 2 --dsw_timeres 1 -d -o rebealance_two_nodes.log -C 127.0.0.1:8050
Comment by Deepti Dawar [ 22/Jan/13 ]
Attaching the functional test results.
This was run against a local 2.0.0 node.
Pass rate is better this time - 92%.
Comment by Deepti Dawar [ 22/Jan/13 ]
For the Hybrid tests - failures are still coming.

Attaching the intermittent log.
Comment by Deepti Dawar [ 22/Jan/13 ]
The error that seems to be problematic in the unit test logs is this one -

'Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.'

Most of the issues coming due to timeout.

Note : that these tests were run against a local cluster. Hence, such problems should not be occurring.
Comment by Michael Nitschinger [ 23/Jan/13 ]
Merged in today, right before the 1.1.1 release.




[JCBC-125] Don't cast view documents to strings Created: 03/Oct/12  Updated: 03/Dec/12  Resolved: 08/Nov/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.1-beta
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: view
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: All


 Description   
When objects are stored as non-strings in Couchbase and then loaded through a View, the SDK currently casts every document to a string. This works fine for JSON documents, but as soon as you want to store serialized objects it breaks.

Implicit casting is not needed in this place (see fix).

 Comments   
Comment by Michael Nitschinger [ 03/Oct/12 ]
Here is a quick sample on how to reproduce it, but I'm adding a correct test case to the code as well (the view used is just the default "emit key and null" view with no reduce).


import com.couchbase.client.CouchbaseClient;
import com.couchbase.client.CouchbaseConnectionFactory;
import com.couchbase.client.protocol.views.Query;
import com.couchbase.client.protocol.views.View;
import com.couchbase.client.protocol.views.ViewResponse;
import com.couchbase.client.protocol.views.ViewRow;
import java.io.IOException;
import java.net.URI;
import java.util.Arrays;
import java.util.Date;
import java.util.Iterator;


public class BinaryviewTest {

  /**
   * @param args the command line arguments
   */
  public static void main(String[] args) throws IOException {

    // Initialize Connection
    CouchbaseClient cb = new CouchbaseClient(
      new CouchbaseConnectionFactory(
        Arrays.asList(
          URI.create("http://192.168.1.105:8091/pools")
        ),
        "default",
        ""
      )
    );

    // Store binary objects
    Object ob1 = new Date();
    cb.set("date1", 0, ob1);

    Object result = cb.get("date1");
    System.out.println(result.getClass());

    Query query = new Query();
    query.setIncludeDocs(true);
    View view = cb.getView("testing", "binary");
    ViewResponse response = cb.query(view, query);

    Iterator<ViewRow> iterator = response.iterator();
    ViewRow row;
    while(iterator.hasNext()) {
      row = iterator.next();
      Object obj = row.getDocument();
      System.out.println(obj.getClass());
    }
    cb.shutdown();
  }
}

When run two times, this raises the exception as described here: http://www.couchbase.com/forums/thread/couchbase-2-view-non-json-docs

I'll update it when I have the fix and test ready.
Comment by Michael Nitschinger [ 03/Oct/12 ]
See http://review.couchbase.com/#/c/21305/
Comment by Michael Nitschinger [ 08/Nov/12 ]
pushed to master, will be available in dp5.




[JCBC-74] Implement observe command Created: 12/Jul/12  Updated: 22/Aug/12  Resolved: 22/Aug/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.1dp2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Raghavan Srinivas (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Add the basic low level observe command

 Comments   
Comment by Raghavan Srinivas (Inactive) [ 22/Aug/12 ]
This is implemented in dp2 and will be enhanced post dp2.




[JCBC-97] documentation needs a discussion on JSON mapping, getting started needs an example of using JSON Created: 10/Aug/12  Updated: 22/Aug/12  Resolved: 22/Aug/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 1.1dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Raghavan Srinivas (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
At one point, I thought we had a discussion of JSON document usage in the getting started and on the web page. It's not there now, but we need to have a discussion about how a Java developer maps their high-level objects into JSON.

 Comments   
Comment by Raghavan Srinivas (Inactive) [ 22/Aug/12 ]
There were a couple of issues. The document changes were reverted back. That's fixed. The Getting Started has been enhanced to persist some JSON documents.




[JCBC-72] View query returns null Created: 05/Jul/12  Updated: 11/Jul/12  Resolved: 11/Jul/12

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1dp
Fix Version/s: 1.1dp2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sharon Barr (Inactive) Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
viewResponse return null, and i can't figure out why:

• I execute my code
CouchbaseClient c = new CouchbaseClient(baseURIs, "default", "");
View view = c.getView("dev_lenders", "country_count");

Query query = new Query();
Stale stale=Stale.OK;
query.setStale(stale);
query.setGroup(true);
ViewResponse viewResponse = c.query(view, query);

• My viewRespons is null.

• Checking the couchdb.1 log, I see:
[couchdb:info] [2012-07-04 21:40:50] [ns_1@127.0.0.1:<0.24865.33>:couch_log:info:39] 10.32.5.29 - - GET /default/_design/dev_lenders/_view/country_count?group=true&stale=ok 200

• Running this URL directly against my server I do get results:
http://10.2.1.12:8092/default/_design/dev_lenders/_view/country_count?group=true&stale=ok
{"rows":[
{"key":null,"value":4171},
{"key":"AE","value":5},
{"key":"AF","value":1},
{"key":"AM","value":2},
...


Not sure what logs i can provide to debug this further. let me know how i can help.

 Comments   
Comment by Sharon Barr (Inactive) [ 05/Jul/12 ]
On slight variation from the above issue, i did mange to get an error from the server via the HTTP request
{"error":"query_parse_error","reason":"Invalid URL parameter 'group' or 'group_level' for non-reduce view."}

but the client still returns null.
Comment by Mike Wiederhold [ 09/Jul/12 ]
http://review.couchbase.org/#change,18094




[JCBC-59] authentication not working correctly with CouchbaseConnectionFactoryBuilder Created: 04/Jun/12  Updated: 06/Aug/12  Resolved: 09/Jun/12

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.0.2
Fix Version/s: 1.0.3
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Raghavan Srinivas (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The use of the CouchbaseConnectionFactoryBuilder like so yields a client that does not authenticate correctly.

CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
CouchbaseConnectionFactory cf = cfb.buildCouchbaseConnection(servers, "default_jOWPgw", "default_jOWPgw", "testing");
CouchbaseClient client = new CouchbaseClient(cf);

 Comments   
Comment by Matt Ingenthron [ 04/Jun/12 ]
http://review.couchbase.org/#change,16765




[JCBC-22] Spring Support for CouchbaseClient Created: 26/Jul/11  Updated: 31/Jan/13  Resolved: 31/Jan/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Mike Wiederhold Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We need to add support for the following Spring Integration Parts:

- General Info on how to use the Client with Spring Beans
- Caching Support through @Cacheable annotations
- Spring Data Support (yet to be defined).

Current progress of the (yet unofficial support) can be tracked here: https://github.com/couchbaselabs/couchbase-spring

 Comments   
Comment by osk [ 19/Nov/12 ]
Need any help on this?
Comment by Michael Nitschinger [ 31/Jan/13 ]
I'm going to close this ticket because it's not part of the couchbase client.

All ongoing efforts can be found here: https://github.com/couchbaselabs/spring-data-couchbase
Comment by Michael Nitschinger [ 31/Jan/13 ]
This is now part of a separate project.




[JCBC-480] replica read with NOT_EXISTS never completes Created: 25/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-151] Client does timeout on connect in specific java environments (was believed to be java7 related). Created: 21/Nov/12  Updated: 21/Jul/14  Resolved: 31/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to JCBC-388 Refactor View Connection Management Resolved

 Description   
People report that the Client does not work with 1.7. Here is a sample stack trace:

2012-11-20 00:29:11.228 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-11-20 00:29:11.240 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@3f757322
2012-11-20 00:29:11.502 INFO com.couchbase.client.ViewConnection: Added localhost/127.0.0.1:8092 to connect queue
2012-11-20 00:29:11.505 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2012-11-20 00:29:11.647 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210
2012-11-20 00:29:12.051 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2012-11-20 00:29:12.060 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: localhost/127.0.0.1:8092 - Session request successful
2012-11-20 00:29:17.111 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [localhost/127.0.0.1:8092(closed)]
java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:68)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:428)
at Example1.main(Example1.java:43)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:80)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:65)
... 2 more
2012-11-20 00:30:12.168 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
2012-11-20 00:30:12.177 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
Disconnected from the target VM, address: '127.0.0.1:65280', transport: 'socket'

Process finished with exit code 0


Also See http://www.couchbase.com/forums/thread/java-asyncconnectionmanager-timed-out-waiting-operation-please-help-console-log-included
And http://stackoverflow.com/questions/13466010/using-java-api-in-scala-to-query-views-in-couchbase-throws-timeout-exception?utm_source=twitterfeed&utm_medium=twitter

 Comments   
Comment by Quinn Slack [ 27/Nov/12 ]
This doesn't appear to be strictly related to Java7. I was able to get similar code to work on Java7. I even simulated loading a Play2 environment with other JARs that could potentially cause conflicts. It's possible that my simulation of the Play2 environment was insufficient and that Play2 does other stuff...

Code at https://github.com/sqs/couchbase-scala-example. Run with:

CBURL=http://localhost:8091/pools CBPASSWORD=mypassword sbt -Dconfig.file=conf/application.conf '~run'

Here is the output on my system that shows it's on Java7 (openjdk) and that shows the expected output from the views:

on java version 1.7.0_09
[info] play - Application started (Prod)
2012-11-27 01:34:27.115 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-11-27 01:34:27.120 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@6267e5a2
2012-11-27 01:34:27.147 INFO com.couchbase.client.ViewConnection: Added localhost/127.0.0.1:8092 to connect queue
2012-11-27 01:34:27.149 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2012-11-27 01:34:27.176 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210
2012-11-27 01:34:27.282 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2012-11-27 01:34:27.288 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: localhost/127.0.0.1:8092 - Session request successful
Res = List(com.couchbase.client.protocol.views.ViewRowNoDocs@4667820f, com.couchbase.client.protocol.views.ViewRowNoDocs@358bcae5, com.couchbase.client.protocol.views.ViewRowNoDocs@6cb59bd9, com.couchbase.client.protocol.views.ViewRowNoDocs@70afb51, com.couchbase.client.protocol.views.ViewRowNoDocs@61f98673)
2012-11-27 01:34:27.396 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
2012-11-27 01:34:27.403 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
Comment by Michael Nitschinger [ 27/Nov/12 ]
Are you running on jdk7 on mac? Also, when you use play 2.1 it should work. Play2-related issues more were because of netty version incompatibilities.
Comment by Quinn Slack [ 27/Nov/12 ]
This is jdk7 on Arch Linux and Play 2.1-RC1.

I tried using a different netty, but no luck. What worked for me was moving calls to "new CouchbaseClient" out of static "object" initializers, since they appeared to be getting called in the Netty I/O thread (I got this exception in new CouchbaseClient: java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread).
Comment by Michael Nitschinger [ 28/Nov/12 ]
Hi Quinn, this issue is a different one! The issue described here is that people seem to find connection timeouts with java 7.
Comment by Quinn Slack [ 28/Nov/12 ]
They may be related--or may just have similar symptoms. I was not getting timeout exceptions, but I was seeing view requests hang indefinitely until I made the fix described above.
Comment by dragos [ 13/Mar/13 ]
Hi Michael, can you help with this error ? I am trying to find a workaround or something to work. This is the only thing which is keeping us from using Couchbase. I also posted here : http://www.couchbase.com/forums/thread/couchbase-connectivity-problem-aws-vpc . As you can see in the times the timeout is received really fast.

2013-03-13 11:40:47.702 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.X.XXX:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-03-13 11:40:52.712 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@5cbfe9d
2013-03-13 11:40:52.840 INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.X.XXX/10.0.X.XXX:11210
2013-03-13 11:40:57.860 INFO com.couchbase.client.ViewConnection: Added 10.0.X.XXX to connect queue
2013-03-13 11:40:57.863 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-03-13 11:40:58.070 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-03-13 11:40:58.077 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.X.XXX:8092 - Session request successful
2013-03-13 11:41:03.113 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.X.XXX/10.0.X.XXX:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
        at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:475)
        at couchbase.training.view.poc.CouchbasePOC.main(CouchbasePOC.java:63)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi dragos,

can you please post the full code where you connect to couchbase? (including factories and such)

Thanks!
Comment by dragos [ 13/Mar/13 ]
Hi Mike, this is the POC client for Couchbase:
package couchbase.training.view.poc;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.TimeUnit;

import org.apache.commons.io.IOUtils;
import org.json.JSONException;
import org.json.XML;

import com.couchbase.client.CouchbaseClient;
import com.couchbase.client.CouchbaseConnectionFactory;
import com.couchbase.client.CouchbaseConnectionFactoryBuilder;
import com.couchbase.client.protocol.views.ComplexKey;
import com.couchbase.client.protocol.views.Query;
import com.couchbase.client.protocol.views.Stale;
import com.couchbase.client.protocol.views.View;
import com.couchbase.client.protocol.views.ViewResponse;
import com.couchbase.client.protocol.views.ViewRow;

public class CouchbasePOC
{
public static void main(String[] args) throws FileNotFoundException, JSONException, IOException
{
System.out.println("Hello from CouchBase");

// Set the URIs and get a client
final List<URI> uris = new LinkedList<URI>();

// Connect to localhost or to the appropriate URI(s)
uris.add(URI.create("http://10.0.7.164:8091/pools"));

final String mappedName="problems";
long start = System.currentTimeMillis();

CouchbaseConnectionFactoryBuilder m_couchbaseConnectionFactoryBuilder = new CouchbaseConnectionFactoryBuilder();
m_couchbaseConnectionFactoryBuilder.setMaxReconnectDelay(10000);
m_couchbaseConnectionFactoryBuilder.setOpQueueMaxBlockTime(100);
m_couchbaseConnectionFactoryBuilder.setOpTimeout(20000);
m_couchbaseConnectionFactoryBuilder.setShouldOptimize(true);
m_couchbaseConnectionFactoryBuilder.setViewTimeout(300000);

CouchbaseConnectionFactory connFactory = m_couchbaseConnectionFactoryBuilder.buildCouchbaseConnection(uris, mappedName, "");

CouchbaseClient client = null;
try
{
client = new CouchbaseClient(connFactory);
}
catch (IOException e)
{
System.err.println("IOException connecting to Couchbase: " + e.getMessage());
System.exit(1);
}

final View view = client.getView(mappedName, mappedName);
final Query query = new Query();

final ComplexKey key = ComplexKey.of("problem",null,null,null,null);
query.setKey(key);
query.setStale( Stale.UPDATE_AFTER );
int counter=0;
try
{
final ViewResponse result = client.query(view, query);
for(ViewRow row : result)
{
counter++;
System.out.println(row.getId());
}
}
catch(java.lang.VerifyError e)
{
System.out.println("Error: " + e.getMessage());
}

System.out.println("time: " + (System.currentTimeMillis()-start) );
System.out.println("documents: " + counter );

client.shutdown(3, TimeUnit.SECONDS);
System.exit(0);
}
}
Comment by dragos [ 13/Mar/13 ]
I hope you would not mind that i called you Mike and not Michael.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hehe no worries.

Can you do me a favor and remove the factory for a second and just use CouchbaseClient() without anything else? Let me know if the problem still persists or not! Thanks
Comment by dragos [ 13/Mar/13 ]
Hi Michael,
First i want to thank you for the quick feed-back. Your owesome !!!

Before using the factory i used the CouchbaseClient(uris, bucketName, "") which is also timing out. I added the factory to be able increase the timeout times for the connection but still the same problem.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hmm okay, so its not the factory (we had some issues with default values in the past, thats why I'm asking).

Can you pin me down your operating system and exact java version you're using?Thanks for your help!
Comment by Michael Nitschinger [ 13/Mar/13 ]
Also, since you're so responsive (I hope we can pin this down finally!).. Can you run the same code with DEBUG log enabled (see https://code.google.com/p/spymemcached/wiki/Logging).. just use Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST); instead of Logger.getLogger("net.spy.memcached").setLevel(Level.FINEST);

Thanks!
Comment by dragos [ 13/Mar/13 ]
This is my test environment for the client which will be similar for production.
OS Ubuntu Server 11.10 x64
java -version result:
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My network infrastructure is like this:
In an AWS virtual private center (VPC) i have a subnet and in this subnet there are 2 machines. The client machine and the Couchbase server. The machines have identical OS and java version as mentioned above. I have open the ports and also checked the network ACL. The traffic is free to go,no restriction. If i am using a rest query with curl for the view i receive the result, the problem is only in the java SDK.
If needed more information please ask. I will gladly offer it.

Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi Dragos,

yes, if you can please rerun the script with DEBUG logging enabled (see last comment).. Thanks very much in advance!
Comment by dragos [ 13/Mar/13 ]
I've setted the logger to finest and this is the error log obtained. Also i tested again the connection with curl and no problemes.

Mar 13, 2013 1:30:14 PM com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
2013-03-13 13:30:14.679 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-03-13 13:30:19.689 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@4e99353f
2013-03-13 13:30:19.810 INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.7.164/10.0.7.164:11210
2013-03-13 13:30:24.828 INFO com.couchbase.client.ViewConnection: Added 10.0.7.164 to connect queue
2013-03-13 13:30:24.830 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-03-13 13:30:25.025 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-03-13 13:30:25.033 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.7.164:8092 - Session request successful
2013-03-13 13:30:30.083 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
        at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:475)
        at couchbase.training.view.poc.CouchbasePOC.main(CouchbasePOC.java:68)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)
        ... 2 more

curl result

curl http://10.0.7.164:8092/problems/_design/dev_problems/_view/problems?stale=false\&connection_timeout=60000\&limit=10\&skip=0
{"total_rows":0,"rows":[
]
}
[4]+ Done
Comment by dragos [ 13/Mar/13 ]
I am available for next steps in debuging. With what can i help next ?
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi, hmm no there should be more output (DEBUG), not just info... you need to copy this whole snippet before your code actually executes:

        // Tell spy to use the SunLogger
        Properties systemProperties = System.getProperties();
        systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");
        System.setProperties(systemProperties);

        Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);

        //get the top Logger
        Logger topLogger = java.util.logging.Logger.getLogger("");

        // Handler for console (reuse it if it already exists)
        Handler consoleHandler = null;
        //see if there is already a console handler
        for (Handler handler : topLogger.getHandlers()) {
            if (handler instanceof ConsoleHandler) {
                //found the console handler
                consoleHandler = handler;
                break;
            }
        }

        if (consoleHandler == null) {
            //there was no console handler found, create a new one
            consoleHandler = new ConsoleHandler();
            topLogger.addHandler(consoleHandler);
        }

        //set the console handler to fine:
        consoleHandler.setLevel(java.util.logging.Level.FINEST);


I guess you're running it from your IDE?
Comment by dragos [ 13/Mar/13 ]
I am building the client on local IDE then uploading it to EC2 test machine. The snipped of code word and now i have a full log.

Mar 13, 2013 1:53:10 PM com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools/default?uuid=44b1e62dd7e65cd38ae7faaabe5ebb64
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools/default/buckets?v=120523822&uuid=44b1e62dd7e65cd38ae7faaabe5ebb64
Mar 13, 2013 1:53:11 PM net.spy.memcached.MemcachedConnection createConnections
INFO: Added {QA sa=/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Updating nodesMap in VBucketNodeLocator.
Mar 13, 2013 1:53:16 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Adding node with address 10.0.7.164:11210.
Mar 13, 2013 1:53:16 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Node added is {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=8}.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=false, w=false, c=true, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=8})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@2c76e369
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection insertOperation
FINE: Added Cmd: 10 Opaque: 1 to {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=1, topRop=null, topWop=null, toWrite=0, interested=8}
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Read 24 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Completed read op: Cmd: 10 Opaque: 1 and giving the next 0 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleInputQueue
FINE: Handling queue
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: No selectors ready, interrupted: false
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: No selectors ready, interrupted: false
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleInputQueue
FINE: Handling queue
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection insertOperation
FINE: Added SASL auth operation to {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL auth operation, toWrite=0, interested=4}
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=false, w=true, c=false, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL auth operation, toWrite=0, interested=4})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=true, w=false, c=false, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL auth operation, topWop=null, toWrite=0, interested=1})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Read 37 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.auth.AuthThread$1 receivedStatus
INFO: Authenticated to 10.0.7.164/10.0.7.164:11210
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Completed read op: SASL auth operation and giving the next 0 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewConnection createConnections
INFO: Added 10.0.7.164 to connect queue
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseClient <init>
INFO: viewmode property isn't defined. Setting viewmode to production mode
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP subscribe
FINE: Subscribing an object for reconfiguration updates com.couchbase.client.CouchbaseClient
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686] OPEN


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is true and Channel State is OPEN.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686, /10.0.7.213:55729 => /10.0.7.164:8091] BOUND: /10.0.7.213:55729


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is /10.0.7.213:55729 and Channel State is BOUND.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686, /10.0.7.213:55729 => /10.0.7.164:8091] CONNECTED: /10.0.7.164:8091


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is /10.0.7.164:8091 and Channel State is CONNECTED.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: STATUS: 200 OK
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: VERSION: HTTP/1.1
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Cache-Control = no-cache
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Content-Type = application/json; charset=utf-8
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Date = Wed, 13 Mar 2013 13:55:42 GMT
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Pragma = no-cache
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Server = Couchbase Server 2.0.0-1976-rel-enterprise
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Transfer-Encoding = chunked
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER:

Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: CHUNKED CONTENT {
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: {"name":"problems","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","streamingUri":"/pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","localRandomKeyUri":"/pools/default/buckets/problems/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/problems/controller/compactBucket","compactDB":"/pools/default/buckets/problems/controller/compactDatabases"},"nodes":[{"couchApiBase":"http://10.0.7.164:8092/problems","replication":0.0,"clusterMembership":"active","status":"healthy","thisNode":true,"hostname":"10.0.7.164:8091","clusterCompatibility":131072,"version":"2.0.0-1976-rel-enterprise","os":"x86_64-unknown-linux-gnu","ports":{"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/problems/stats","directoryURI":"/pools/default/buckets/problems/statsDirectory","nodeStatsListURI":"/pools/default/buckets/problems/nodes"},"ddocs":{"uri":"/pools/default/buckets/problems/ddocs"},"nodeLocator":"vbucket","autoCompactionSettings":false,"fastWarmupSettings":false,"uuid":"7675b7791efe17220792c2b41ff6c824","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.0.7.164:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 3066
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: 0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 2048
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: ],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 3072
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: 1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"bucketCapabilitiesVer":"","bucketCapabilities":["touch","couchapi"]}
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 361
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketMonitor logFiner
FINER: Getting server list returns this last chunked response:
{"name":"problems","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","streamingUri":"/pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","localRandomKeyUri":"/pools/default/buckets/problems/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/problems/controller/compactBucket","compactDB":"/pools/default/buckets/problems/controller/compactDatabases"},"nodes":[{"couchApiBase":"http://10.0.7.164:8092/problems","replication":0.0,"clusterMembership":"active","status":"healthy","thisNode":true,"hostname":"10.0.7.164:8091","clusterCompatibility":131072,"version":"2.0.0-1976-rel-enterprise","os":"x86_64-unknown-linux-gnu","ports":{"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/problems/stats","directoryURI":"/pools/default/buckets/problems/statsDirectory","nodeStatsListURI":"/pools/default/buckets/problems/nodes"},"ddocs":{"uri":"/pools/default/buckets/problems/ddocs"},"nodeLocator":"vbucket","autoCompactionSettings":false,"fastWarmupSettings":false,"uuid":"7675b7791efe17220792c2b41ff6c824","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.0.7.164:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"bucketCapabilitiesVer":"","bucketCapabilities":["touch","couchapi"]}
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: Received an update, notifying reconfigurables about a com.couchbase.client.vbucket.config.Bucketcom.couchbase.client.vbucket.config.Bucket@4cf3fdc4
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: It says it is problems and it's talking to /pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseConnection reconfigure
FINE: Node 10.0.7.164/10.0.7.164:11210 will stay in cluster config after reconfiguration.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.VBucketNodeLocator updateLocator
FINE: Received updated configuration with insignificant changes.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: Received an update, notifying reconfigurables about a com.couchbase.client.vbucket.config.Bucketcom.couchbase.client.vbucket.config.Bucket@fadb83cf
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: It says it is problems and it's talking to /pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseConnection reconfigure
FINE: Node 10.0.7.164/10.0.7.164:11210 will stay in cluster config after reconfiguration.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.VBucketNodeLocator updateLocator
FINE: Received updated configuration with insignificant changes.
Mar 13, 2013 1:53:21 PM com.couchbase.client.http.AsyncConnectionManager processConnectionRequests
INFO: Opening new Couchbase HTTP connection
Mar 13, 2013 1:53:21 PM com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback completed
INFO: /10.0.7.164:8092 - Session request successful
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewNode$EventLogger connectionOpen
FINE: Connection open: [/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionTimeout
SEVERE: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionClosed
FINE: Connection closed: [10.0.7.164/10.0.7.164:8092(closed)]
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi, thanks for the log..

interestingly, there is nothing unusual in this.. I'll investigate further and come back to you if I need more info..

thanks so far!
Comment by dragos [ 13/Mar/13 ]
Thank you for spending time on this. Looking forward for a fix :) .
Comment by Michael Nitschinger [ 13/Mar/13 ]
interestingly, the connection is open and then times out...

INFO: /10.0.7.164:8092 - Session request successful
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewNode$EventLogger connectionOpen
FINE: Connection open: [/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionTimeout
SEVERE: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionClosed
FINE: Connection closed: [10.0.7.164/10.0.7.164:8092(closed)]

this sounds highly suspicious to me, maybe this is a bug in apache httpcore-nio...
Comment by dragos [ 13/Mar/13 ]
The time out is displayed after 5 seconds. As you suggested i removed the factory and the detailed logged above i obtained with using CouchbaseClient(uris, mappedName, ""). Do you have some way of testing the httpCore-nio ? Should i try to look for latest version of httpCore-nio.. ?
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi,

yes you could try that.. I ran the test suite with 4.2.3 which is the latest one and there were no errors, so our code should work with it. You need to exchange the httpcore and httpcore-nio with the latest 4.2.3 jars

I don't think this is 100% the issue, but it helps us get rid of one more factor!

Thanks!
Michael
Comment by Michael Nitschinger [ 13/Mar/13 ]
Okay, the 5 second period can come from this:

 HttpParams params = new SyncBasicHttpParams();
      params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 5000)
          .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 5000)
....


SO_TIMEOUT is: Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).

and CONNECTION_TIMEOUT is: Determines the timeout in milliseconds until a connection is established.


Dragos, can it be the case that it takes more than 5 seconds to return a result when using curl or so? That would explain the "fail fast" timeout here.. not saying that these times are okay this way, just to find the root cause..

Thanks!
Comment by dragos [ 13/Mar/13 ]
Hi Michael,
I've made a test with httpcore-nio-4.2.jar and with httpcore-4.2.1.jar and your intuition is good. The httpcore is not the problem because the timeout is still there. With curl the results is back fast, in less then half a second. So the problem must be somewhere in the java SDK in the way the view is retrieved.
Comment by dragos [ 13/Mar/13 ]
I've moved the Couchbase server to the same machine as the client to have all in same environment for testing. Now all is working fine but it is not possible for our product to stay on the same machine as Couchbase server for obvious reasons. In production we will have more then one app machine which we will try to access the Couchbase cluster.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi Dragos,

okay so its definitely something with those timeouts.. The thing is we can increase them for sure, but the question is if it would work for your application given the long network latency - and if your application stays performant that way.

When you're running curl on a request, how long does it take?
Comment by dragos [ 14/Mar/13 ]
The curl request is very fast (a few milliseconds, and less then 0.5 seconds i am sure) and the network latency it is not a problem. Some how the couchbase client is taking to much time connecting. When it need to connect to a network machine in the same lan it takes more then 10 seconds. I experimented this in EC2 environment and on my local machine with a VM. If couchbase is local with the test app, the couchbase client is connecting fast.
Do you have any points on how a connection should be setup for couchbase client ? I am trying to use the factory builder and the connection factory from java sdk but still with no success.
Comment by Tim Smith (Inactive) [ 14/Mar/13 ]
Dragos,

I can think of two separate things to try. One is to increase the SO_TIMEOUT to 8000, CONNECTION_TIMEOUT to 12000. Identify what's being hit.

Probably more effective would be to use Wireshark to capture all traffic on the client machine, and see what is really happening. Start the capture (with tshark command-line tool, for example), then run the test client, let it time out, then run the curl command, let it succeed, then stop the capture. Gzip the capture data and attach it to this issue.

Tim
Comment by dragos [ 14/Mar/13 ]
Hi Tim,
I've started some load tests to be able to evaluate Couchbase, so this means i moved Couchbase server on the same machine as the load test client. I tried to use a pool of couchbase clients always connected to the server, and after a while the test failed because couldn't create more connections. From my initial research i seen that couchbase client is managing his own connection can you tell me how this is done or what are the policies of shutting down connections ? Can you point me to some best practices for couchbase client connectivity ?
Comment by Tim Smith (Inactive) [ 14/Mar/13 ]
I will handle that question about connection pooling, etc., separately, since it is unrelated to this bug about timeouts.

On this specific bug report, the current state as I understand it is that: curl returns very quickly with the correct results. The Java client times out after 5 seconds without returning the results.

My suggestion is to set up a very simple Java client that performs a single view query, and to get a packet capture of both curl and the Java client, to identify what the two are doing differently and why the curl succeeds but the Java client fails.
Comment by Jahangir Zinedine [ 27/Mar/13 ]
Hi Michael,

Any update on this issue?
I faced the same issue and here is my environment:
JRuby 1.7.3
Oracle JDK 1.7(with 1.6 the same thing happened)
Couchbase 2.0.1
Java SDK 1.1.3(with 1.1.4 the same thing happened)
And latest netty and http-core and other jar files.

Appreciate your help.

Thanks,
Jani
Comment by dragos [ 01/Apr/13 ]
Hi All,
Finally after a long time i found the workaround needed for the java client to not timeout. As has been suggested by the Couchbase team to use a host name for the Couchabase server i applied the same fix but to the client machine. Meaning on the client machine were the java Couchbase client was timing out ( in Amazon VPC ) we added a hostname for the Couchbase server. Our test URI from http://10.0.7.164:8091/pools has become http://couchbase:8091/pools. This workaround is also fixing a 10 seconds connectivity for the client.

On Ubuntu the hostname file is at: /etc/hosts
On Windows 7 the hostname file is at: C:\Windows\system32\drivers\etc\hosts

Example of host name entry in client machine:
10.0.7.146 couchbase
Comment by Michael Nitschinger [ 02/Apr/13 ]
Thanks very much dragos for tracking this down!

Does this mean that in your environment after the change, you're not seeing any performance impact anymore? Everything works as expected and performant?

Thanks!
Comment by dragos [ 02/Apr/13 ]
Hi Michael,
In this moment the client does not time out and the time taken by java couchbase client to connect takes around 1.5 seconds from 10 seconds which we can consider a normal behavior. As for performance with this workaround we can continue the POC and evaluate the performance. As a first impression i can say that we see a good speed but i will keep my reservation until final POC is tested and we can achieve a full load test.

What i want to emphasize that this is a welcomed workaround but it is not a solution on a longer term for our cloud production site. In a scenario where we need to create new machines and maybe new couchbase clusters it involves manual or specific automatic changes to the hosts file and of course extra managing of this. Bottom line is that we are waiting the fix on the couchbase java sdk 1.1.5 which will make this workaround obsolete.

Also i feel the need to mention that this workaround was found in collaboration with the Couchbase team which provided the basic information for the workaround to be found, so thank you guys.
Comment by Michael Nitschinger [ 02/Apr/13 ]
Hi Dragos,

I'm not sure if there is a thing that we can fix inside the client when this is a DNS/hostname resolution issue with the OS and amazon?

We would definitely need to investigate further, but all that the client does is try to use the hostname/IP provided during bootstrap. If nothing comes back or the host can't be found, we can't do much about it. Anyway, I think we need to do some more investigation here to really see whats the problem.
Comment by aboyev [ 09/Sep/13 ]
Hello, Michael!
I have the same problem. Here is my setup:
- Couchbase 2.1.1
- Java SDK 1.1.9
- Java 1.6.0_27 (and also tried 1.7)
[ERROR] getView | Exception while doing getView: Timed out waiting for operation
[ERROR] Exception while doing getView: I/O reactor has been shut down
It surely has something to do with the client, because when I launch second client it works properly for a while.
Also, it seems to happen under high loads more frequently
Comment by gaurang jhawar [ 24/Oct/13 ]
How did u change the "Meaning on the client machine were the java Couchbase client was timing out ( in Amazon VPC ) we added a hostname for the Couchbase server. Our test URI from http://10.0.7.164:8091/pools has become http://couchbase:8091/pools. This workaround is also fixing a 10 seconds connectivity for the client." in AWS .. I have EC2 instance for which i want to set the hostname and connect through it like you did... Can you tell me the exact steps for that
Comment by Michael Nitschinger [ 25/Oct/13 ]
Hi Gaurang,

are you also exhibiting a delay on connect? Or during operations? I think these two may be related but should be looked at separately and defined. What environment, client, workload and so on are you running?
Comment by gaurang jhawar [ 25/Oct/13 ]
My env is JDK 1.7.0_09
Client is my local machine ...

Couchbase server installed on aws ..

Reconnecting due to failure to connect to {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:423)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:261)
at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
2013-10-24 22:14:01.893 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1.
2013-10-24 22:14:05.903 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2013-10-24 22:14:09.632 INFO com.couchbase.client.ViewConnection: Added 172.31.9.124 to connect queue
Comment by Michael Nitschinger [ 25/Oct/13 ]
One thing to note here is that this is not a good idea. You have "the internet" between your computer (client) and the server which adds latency. Can you try running the application also in an AWS instance and see if the error goes away?
Comment by gaurang jhawar [ 25/Oct/13 ]
I gave the elastic amazon IP in the url and also tried what dragos tried but that also didn`t work for me .. I made ALL TCP ports on firewall ON ... And I am not sure why it reconnects to the IP or fails to connect and load data in the bucket.

But its the same case as dragos ... for production that`s not viable .. since not everything can be on AWS.

Also I can connect to the same instance via telnet and http(browser) .. I don`t know why I am facing this issue with Java 1.7.

"2013-10-24 22:14:01.893 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1.
2013-10-24 22:14:05.903 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2013-10-24 22:14:09.632 INFO com.couchbase.client.ViewConnection: Added 172.31.9.124 to connect queue
2013-10-24 22:14:09.632 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
in INIT **************.
Here is the entity set : users
2013-10-24 22:14:09.868 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 0.
30 Seconds: Load is in progress
2013-10-24 22:14:09.993 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-24 22:14:12.037 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@6dae04e2
2013-10-24 22:14:12.037 INFO com.couchbase.client.CouchbaseConnection: Reconnecting due to failure to connect to {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection timed out: no further information"

NOTE here it tried to reconnect to the client and says its inactive and I am not sure if this is a firewall issue or something else.
Comment by Michael Nitschinger [ 25/Oct/13 ]
Can you provide debug log information for the connect process? Maybe this helps pinning down where the issue is. (you can read here how to do it if you don't know already) :http://nitschinger.at/Logging-with-the-Couchbase-Java-Client

thanks
Comment by gaurang jhawar [ 25/Oct/13 ]
http://s000.tinyupload.com/index.php?file_id=46832814372422198765
Comment by Michael Nitschinger [ 25/Oct/13 ]
Thanks, can you also share the code you use?

In the code it looks like you are creating the CouchbaseClient object twice.
Please note that the bootstrapping succeeds (!) it is different from the ticket reported here.

Are you sure port 11210 is open from client to server? It looks like something is blocking it, http seems to work fine.
Comment by gaurang jhawar [ 25/Oct/13 ]
http://s000.tinyupload.com/index.php?file_id=34200463437173886967



ya the port is opened.

http://s000.tinyupload.com/index.php?file_id=06508245784961590671

Both inbound / outbound on aws as well as my local machine firewall ports are open
Comment by gaurang jhawar [ 26/Oct/13 ]
Any updateS?
Comment by couchbase-user [ 27/Oct/13 ]
This is a nightmare, can't believe no solution has been found yet for over a year now.....
Comment by Michael Nitschinger [ 28/Oct/13 ]
Hi couchbase-user, are you also running app/db over the internet or on a specific ec2 environment?
Comment by couchbase-user [ 28/Oct/13 ]
hey, actually I'm running them all on a local small network.
I am using ubuntu 12.4 on all machines and running java application on headnode to get view data and store them locally.
Comment by Michael Nitschinger [ 28/Oct/13 ]
Hm, I wonder why you are hitting this, I'm not sure its related.. do you get the same exception as the top of the ticket here? Can you provide debug logs so we can see at which point it is happening?

Maybe we need to tweak socket timeouts.
Comment by couchbase-user [ 28/Oct/13 ]
hey, here is my execution log:

28-Oct-2013 09:29:09 com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
2013-10-28 09:29:09.618 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.100:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:09.620 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:09.620 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.102:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:14.635 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@7f09fd93
2013-10-28 09:29:14.635 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@79a5f739
2013-10-28 09:29:14.636 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@68e6ff0d
2013-10-28 09:29:14.656 INFO com.couchbase.client.ViewConnection: Added Cloud-assistant.local to connect queue
2013-10-28 09:29:19.663 INFO com.couchbase.client.ViewConnection: Added 192.168.0.102 to connect queue
2013-10-28 09:29:19.666 INFO com.couchbase.client.ViewConnection: Added 192.168.0.100 to connect queue
2013-10-28 09:29:19.667 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-10-28 09:29:19.788 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-10-28 09:29:19.794 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /192.168.0.102:8092 - Session request successful
2013-10-28 09:29:24.815 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [192.168.0.102/192.168.0.102:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
    at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:483)
    at HelloWorld.Export_Couchbase_View(HelloWorld.java:85)
    at HelloWorld.main(HelloWorld.java:287)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
    at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)

I thought this is exactly the same as the error at the top of this thread....
Comment by Michael Nitschinger [ 28/Oct/13 ]
In your logs I can see that 192.168.0.101 is changed to Cloud-assistant.local. Could it be that this is a DNS issue in your environment?
Also please make sure that ports 8091, 8092 (!) and 11210 are reachable on each node.

Both the socket and connection timeout for views are set to 5 second which is already "quite high" if you want to run a real workload over it.
Comment by couchbase-user [ 28/Oct/13 ]
I'm not sure what the DNS has to do with it or how to check it, something to note here is that sometimes it works fine (very rare).
I have also just tried to change the switch but still the same problem.
I'm working on investigating the DNS and firewall issue. thanks
Comment by Michael Nitschinger [ 28/Oct/13 ]
Okay, if you go directly to the view url, how long does it take you to? like with curl... so we can see how far you are over the 5 seconds and if you are under it then there's something else going on.
Comment by couchbase-user [ 28/Oct/13 ]
I have added different servers names to hosts files in all servers and disabled the firewall on all of them and now it seems to work fine.
Thanks a lot for your help Michael.
Comment by Michael Nitschinger [ 28/Oct/13 ]
perfect, good to know it works now.
Comment by gaurang jhawar [ 28/Oct/13 ]
Any updates on my issue Michael Nitschinger. Thanks
Comment by Michael Nitschinger [ 29/Oct/13 ]
Since we fixed the suspected issue with "couchbase-user" in kind of the same way,
ca you first verify that

- All ports are reachable from all the boxes (11210, 8091, 8092)
- DNS settings are configured properly, this is especially important in EC2

If you look through the thread here, some of the people reporting this thing fixed once they made sure their environmental settings were okay. Can you double check?
Comment by gaurang jhawar [ 31/Oct/13 ]
Couchbase-user how did u configure hostname . can u tell me a step by step approach .. dont give me links to couchbase documentation that didn`t help ..

Michael .. the ports are open I am able to ping .. it connects via java and everything but the timeout is quite frequent ..



C:\Users\gaurang\Downloads>tcping 54.219.141.78 8091

Probing 54.219.141.78:8091/tcp - Port is open - time=21.692ms
Probing 54.219.141.78:8091/tcp - Port is open - time=19.357ms
Probing 54.219.141.78:8091/tcp - Port is open - time=15.414ms
Probing 54.219.141.78:8091/tcp - Port is open - time=16.225ms

Ping statistics for 54.219.141.78:8091
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milli-seconds:
     Minimum = 15.414ms, Maximum = 21.692ms, Average = 18.172ms

C:\Users\gaurang\Downloads>tcping 54.219.141.78 11210\

Probing 54.219.141.78:11210/tcp - Port is open - time=18.236ms
Probing 54.219.141.78:11210/tcp - Port is open - time=23.513ms
Probing 54.219.141.78:11210/tcp - Port is open - time=1218.881ms
Probing 54.219.141.78:11210/tcp - Port is open - time=14.712ms

Ping statistics for 54.219.141.78:11210
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milli-seconds:
     Minimum = 14.712ms, Maximum = 1218.881ms, Average = 318.836ms

Comment by CathodTech [ 13/Dec/13 ]
We have been facing the same issue for couples of months - with Couchbase on RHEL, and we have just found out the solution that works for us.

We have 2 Tomcat nodes running; 1 with Java 1.7 and another with 1.6. Couchbase was deployed on another 4 servers in the same network. We simply defined all Couchbase nodes in /etc/hosts file in both Tomcat nodes as you normally would (with x.x.x.x hostname), then it magically works for us - without having to restart Tomcat, nor Couchbase nodes.
Comment by Michael Nitschinger [ 13/Dec/13 ]
Thanks for pointing this out, I'll see if I can get it into the official docs to aid other people. To me this really looks like java-related and network-related. Interestingly only a few people are hitting it.
Comment by CathodTech [ 14/Dec/13 ]
For the issue I faced, it looks more like Java API related. Perhaps, something to do with DNS (e.g., something like gethostbyname() / gethostbyaddr()), and how the API handles the case when IP has no hostname. Note that we only use IP address to add nodes into Couchbase cluster and Java on other servers also connect to Couchbase with IP address, not hostname.

From the trace, we found that java code has no problem connecting (via connection pool) with Couchbase nodes that has IP/hostname pair in /etc/hosts. But for Couchbase nodes that were not defined in /etc/hosts, it attempted to connect, then the connection dropped, and returned timeout issue.

My team also found that later when he restarted java to re-deploy, Couchbase connection attempts at start-up were much faster (on both Java 1.6 and 1.7).

For this issue, we didn't face it when deploying Java and Couchbase on the same server as, perhaps, it is common to have "127.0.0.1 localhost" defined in /etc/hosts.

We hope that the solution we found would help you and others as well. Not sure if others are facing the same scenario, though.
Comment by David Borsos [ 18/Dec/13 ]
I've ran into a very similar/the same situation with the latest couchbase (2.2.0) server and java client (1.2.3), I thought I share my experiences.

The problem I faced was when trying to call CouchbaseClient.getView(). This apparently goes to one node in the couchbase cluster and queries some view metadata over HTTP (roughly). However I got "ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out" and that pretty much killed my client there. Seeing the solutions here with the hosts file workaround, I tried that and it indeed worked, but it's not really a suitable solution for us, so I decided to see into this a little bit more. Here is what I found.

(Disclaimer: this worked for me, and although I found other ways around this they may not work for you, and I did these attempting to solve this particular issue and didn't really care about anything else these changes may break, so be careful!)

(0) Baseline, the environment specific thing that actually causes the issue is that call java api call InetAddress.getHostFromNameService(InetAddress addr, boolean check) returns very slowly if there's no hostname to be found for a given IP. I have no idea why is that in my environment, I suppose could be because of a bunch of different reasons. Slow here means that it takes 4-5 secs for me for this call to return with an unknown hostname

This comes into the picture when the couchbase client tries to get a view. It uses Apache Http client to do this, and the following things happen:
(1) When the request for the view gets created, eventually we get to BaseIOReactor.execute which instantiates SessionHandle, and registers the time when this happened (this will cause the timeout)
(2) Somewhere in the request processing, we actually DO hit getHostFromNameService. More specifically this happens because a RequestTargetHost interceptor runs and this injects the "Host" header into the HTTP request.
(3) some time after (2) we hit a timeout check which compares the timestamp recorded in (1) and the current time, obviously resulting in a timeout and the error in the client if (2) takes a long time as indicated

The ideal solution would obviously be to fix whatever is wrong with my DNS configs, but I'm not (yet) sure what's wrong (running Ubuntu 13.10 btw). And also make sure that our live environments won't have the same issue...

One workaround I found that might be generally usable is to override the JVM's DNS resolver, by starting the client with: -Dsun.net.spi.nameservice.provider.1=dns,sun This worked for me, but I don't know what side-effects it may have (generally this may not be a great idea). Additionally this resulted in a significant increase of startup speed as the couchbase client seems to do a lot of ip->hostname lookups when it's starting (are these really necessary?)

Also I'd like to ask from the Couchbase folks if it's really necessary to do this reverse DNS lookup for every HTTP request. I cloned your client from git, and removed the RequestTargetHost interceptor from these calls; that also solved the timeout problem, and all seems to work fine. I didn't do exhaustive testing on it, and this, too, can be environment specific - running my couchbase cluster in local VMs at the moment.

Thanks.
Comment by Michael Nitschinger [ 19/Dec/13 ]
Hi David,

thanks for the heads-up, that gives me a better idea of whats going on. I basically revamped the whole view connection management for the next 1.3 release which hopefully should address this to. Would you want to try it out based on a preliminary build?

The thing is that you can't remove the RequestTargetHost, but it only does the DNS lookup if HTTP.TARGET_HOST is not set, but we can infer that from our configuration. So the new code basically sets it before the request is put in the queue, once we've determined which node to ask, then it should be skipped.

I already have that locally, so if you want to try it out we can quickly see if it fixes the issue.
Comment by Michael Nitschinger [ 19/Dec/13 ]
The issue will be fixed with the overall changeset of jcbc-388, which will appear in - as it stands now - 1.3, slated for jan 2014.
Comment by David Borsos [ 19/Dec/13 ]
Michael,

That sounds great. I'd be happy to help you verify your new code; send it over or let me know how can I get it, I'll switch to it and see if it solves the problem. If we can get the new version in early January (I saw it's scheduled for the 7th) that's cool too, we won't be doing much during the holidays anyway :)

Thanks
Comment by Sergii [ 19/Dec/13 ]
Hi Michael,

i have run into the same issue with couchbase 2.2 on mac - Java 1.7

Can i have the changed jar to try the fix?

Thanks
Comment by Michael Nitschinger [ 31/Dec/13 ]
A fix for this has been merged into master and will be available in the next (1.3.0) release.
Comment by Michael Nitschinger [ 31/Dec/13 ]
The new code minimizes dns lookups to workaround the issue, but the environment still needs to be fixed properly :)
Comment by Oded Hassidi [ 21/Jul/14 ]
Was this fixed, since we are having issues with timeot on Java SDK v1.4.x, and we are using JDK 7?
This is not happen all the time, if I run a thread that runs many request it occur every 2-4 min
Comment by Michael Nitschinger [ 21/Jul/14 ]
This issue happened during startup, are you experiencing issues during startup? Where do you get the timeouts?
Comment by Oded Hassidi [ 21/Jul/14 ]
Thanks for the quick response.
The timeout are not at startup it happens during runtime. every 2-3 minutes I get the followed:

SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:778)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:65)
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:49)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
... 45 more




[JCBC-472] Implement support for snappy compression where supported Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-471] Implement HELLO command Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-473] Flush support Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-456] view query hangs indefinitely, prevents client shutdown too Created: 06/May/14  Updated: 12/May/14  Resolved: 12/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Mac OS X, Java 1.7.0_55 (I think)


 Description   
Grab the Developer Day code at https://github.com/couchbaselabs/DeveloperDay

Update the client to version 1.4.0.

Run example #9.

For me, it hangs after the last view request. If I downgrade to 1.3.2, it does not hang.

 Comments   
Comment by Matt Ingenthron [ 12/May/14 ]
verified fixed in 1.4.1




[JCBC-451] Title on getting started refers to 1.3 Created: 18/Apr/14  Updated: 18/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.2
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: gettingstarted
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: http://www.couchbase.com/communities/java/getting-started


 Description   
The title of the page is Couchbase Java Client Library 1.3. It should be 1.4.

 Comments   
Comment by Michael Nitschinger [ 18/Apr/14 ]
thanks!




[JCBC-421] Query.setLimit() performance problem Created: 26/Feb/14  Updated: 04/May/14  Resolved: 15/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 1.3.2
Fix Version/s: 1.4.0-dp2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Philip Luppens Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: OSX 10.9.1, JDK 1.7.0_45-b18, Couchbase 2.1.1 CE

Issue Links:
Gantt: start-finish
is triggered by SPY-127 Address performance of try/catch in i... Resolved

 Description   
I've encounter a substantial slowdown in my code when using a view in combination with a query. It seems that setting the limit (other setters have not been tried, but are likely to show similar behaviour) will eventually call net.spy.memcached.util.StringUtils#isJsonObject(). In 2.10.5 and earlier, this method will use 'new Integer()/catch NumberFormatException' to test if the passed argument can be parsed to an Integer object. However, the number of thrown and caught exceptions was severely limiting performance on my setup.

Removing the setLimit() call on my Query object resulted in a threefold increase in performance (from 600 to 1900 rps).

Original SpyMemcached issue: http://code.google.com/p/spymemcached/issues/detail?id=298

 Comments   
Comment by Michael Nitschinger [ 26/Feb/14 ]
Thanks, I'm looking into it for the 1.4.0 release
Comment by Michael Nitschinger [ 26/Feb/14 ]
What setLimit() value are you using?

While looking at the issue, even if you go in there with a value of 1 new Integer() should not throw an exception and return true?
Maybe it comes from other params in your Query part?
Could you potentially upload that profile data somewhere that I can take a look at it?
Comment by Michael Nitschinger [ 26/Feb/14 ]
http://review.couchbase.com/#/c/33930/
Comment by Philip Luppens [ 26/Feb/14 ]
Well, here is the code (not much to it):

val query: Query = new Query()
query.setKey(playerKey)
query.setLimit(1)

That is all ...
Not sure if I can get the snapshot uploaded - but I can get you a screenshot of the profiling session: http://nsesa.org/uploads/screenshots/profiler.png
Comment by Michael Nitschinger [ 03/Mar/14 ]
Alright, I did a complete rewrite of the Query internals, it is much faster now. Will for sure get this into 1.4.0 final, I'll attach some JMH benchmark results in a second.
Comment by Philip Luppens [ 03/Mar/14 ]
Thank you very much, Michael. Looking forward to the 1.4 release.
Comment by Michael Nitschinger [ 03/Mar/14 ]
Resembling your use case (note that I named old and new query objects for simplicity reasons):

    @GenerateMicroBenchmark
    public void oldWithLimitAndKey(BlackHole bh) {
        bh.consume(new OldQuery().setLimit(100).setKey("user-michael").toString());
    }

    @GenerateMicroBenchmark
    public void newWithLimitAndKey(BlackHole bh) {
        bh.consume(new NewQuery().setLimit(100).setKey("user-michael").toString());
    }

Benchmark Mode Samples Mean Mean error Units
b.Benchmark.newWithLimitAndKey thrpt 10 1234.499 10.420 ops/ms
b.Benchmark.oldWithLimitAndKey thrpt 10 235.442 5.141 ops/ms


... in your case this is a 500% perf increase on the class (1.2k ops/ms instead of 235 ops/ms).





[JCBC-417] Need asyncGetFromReplicaS() method Created: 19/Feb/14  Updated: 25/Feb/14  Resolved: 25/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.0-dp1
Security Level: Public

Type: New Feature Priority: Major
Reporter: Larry Liu Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There is no asyncGetFromReplicaS() method currently which can return CAS value like the asyncGetS(). This will help passing the CAS value in subsequent writes using CAS.

 Comments   
Comment by Michael Nitschinger [ 20/Feb/14 ]
looking into it for 1.4, it would be asyncGetsFromReplica :)




[JCBC-414] typo in release notes for 1.3.2 Created: 11/Feb/14  Updated: 20/Feb/14  Resolved: 20/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
"if a custom execturo is passed in."




[JCBC-409] clarify behavior or *getFromReplica in javadoc Created: 03/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.0, 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I've seen questions a few times on how replica read behaves w.r.t. multiple replicas and recovery of the active location.

IIRC, the getFromReplica() will give you the first responder and the asyncGetFromReplica() will give you a future which has all the responses? The javadoc doesn't indicate this though, so it'd be great if we can clarify.




[JCBC-408] javadoc for ComplexKey class formatted incorrectly Created: 02/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Some of the code examples in the javadoc are not formatted correctly.

See:
http://www.couchbase.com/autodocs/couchbase-java-client-1.3.1/com/couchbase/client/protocol/views/ComplexKey.html




[JCBC-463] Java client occasionally leaks (non-daemon) netty IO threads after shutdown Created: 27/May/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Task Priority: Major
Reporter: Dave Engberg Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We're using CouchbaseClient v1.4 from a Java server with moderately high concurrency. When we shutdown the Couchbase client, it usually comes down cleanly, but we occasionally see lingering Netty threads
{noformat}
"New I/O worker #1" prio=10 tid=0x00000000014a3000 nid=0x58bd runnable [0x00007fc7c2f88000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059a590988> (a sun.nio.ch.Util$2) - locked <0x000000059a5909a0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a5af320> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:223) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) "Memcached IO over {MemcachedConnection to /10.6.70.3:11210 /10.6.70.2:11210 /10.6.70.1:11210}" prio=10 tid=0x00007fc7dcd48800 nid=0x58b8 runnable [0x00007fc7c38f1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059e331fe0> (a sun.nio.ch.Util$2) - locked <0x000000059e331ff8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a513d90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:222) at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158)
{noformat}

This prevents clean Java server shutdown, and requires an aggressive "kill -9" to clean up. We figured out how to make the second thread start as 'daemon', but the "New I/O worker" thread can't be converted to daemon without gutting the client libs pretty heavily.

I haven't been able to reproduce this in isolated unit tests. We (Evernote) run around 500 Java7+Tomcat servers that receive a lot of activity over the course of a week, and see this problem come up in the wild around 20% of the time when we try to shut down. So it's a bit hard to narrow down the exact conditions that cause the thread leakage.

Our current workaround:
{noformat}
--- /tmp/BucketMonitor.java 2014-05-16 10:04:58.000000000 -0700
+++ src/main/java/com/couchbase/client/vbucket/BucketMonitor.java 2014-05-16 10:04:16.000000000 -0700
@@ -97,13 +97,28 @@
     this.configParser = configParser;
     this.host = cometStreamURI.getHost();
     this.port = cometStreamURI.getPort() == -1 ? 80 : cometStreamURI.getPort();
- factory = new NioClientSocketChannelFactory(Executors.newCachedThreadPool(),
- Executors.newCachedThreadPool());
+ factory = new NioClientSocketChannelFactory(newThreadPool(),
+ newThreadPool());
     this.headers = new HttpMessageHeaders();
       this.provider = provider;
   }
 
   /**
+ * Creates an executor based on a simple thread pool that only
+ * uses 'daemon' threads.
+ */
+ private java.util.concurrent.Executor newThreadPool() {
+ return Executors.newCachedThreadPool(
+ new java.util.concurrent.ThreadFactory() {
+ public Thread newThread(Runnable r) {
+ Thread t = new Thread(r);
+ t.setDaemon(true);
+ return t;
+ }
+ });
+ }
+
+ /**
    * Take any action required when the monitor appears to be disconnected.
{noformat}

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
I've got a similar report with the memcache IO thread, maybe this is related. I'll dig into it further and update the progress.
Comment by Michael Nitschinger [ 02/Jun/14 ]
I started work in both spy and jcbc to tackle this individually and harden the shutdown procedures in both.

http://review.couchbase.org/#/c/37726/
http://review.couchbase.org/#/c/37724/




[JCBC-448] TU1 Upgrade scenario | ConcurrentModificationException in memcached connection while checking timeout Created: 07/Apr/14  Updated: 10/Apr/14  Resolved: 10/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
[445.40 INFO] (SDKD log:137) ESC[39mApr 7, 2014 5:40:59 AM net.spy.memcached.MemcachedConnection checkPotentiallyTimedOutConnectionESC[0;39m
[445.40 INFO] (SDKD log:137) ESC[39mWARNING: Retrying selector keys after ConcurrentModificationException caughtESC[0;39m
[445.40 INFO] (SDKD log:137) ESC[39mjava.util.ConcurrentModificationExceptionESC[0;39m
[445.99 INFO] (LineGobbler doFilter:115) ESC[39m+++ Following exception has internal ID: 7ESC[0;39m
[445.99 INFO] (SDKD log:137) ESC[39m at java.util.HashMap$HashIterator.nextEntry(HashMap.java:810)
        at java.util.HashMap$KeyIterator.next(HashMap.java:845)
        at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1026)
        at net.spy.memcached.MemcachedConnection.checkPotentiallyTimedOutConnection(MemcachedConnection.java:496)
        at net.spy.memcached.MemcachedConnection.handleOperationalTasks(MemcachedConnection.java:428)
        at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:417)
        at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:299)ESC[0;39m
[445.99 INFO] (SDKD log:137) ESC[39mApr 7, 2014 5:40:59 AM net.spy.memcached.MemcachedConnection handleIOESC[0;39m
[445.99 INFO] (SDKD log:137) ESC[39mINFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@65b4abcbESC[0;39m
[446.00 INFO] (SDKD log:137) ESC[39mApr 7, 2014 5:40:59 AM com.couchbase.client.CouchbaseConnection addOperationESC[0;39m

 Comments   
Comment by Deepti Dawar [ 10/Apr/14 ]
This is coming because the retry logic has been added in spy for the potential timeout.




[JCBC-435] update README info on maven repos Created: 21/Mar/14  Updated: 01/Jun/14  Resolved: 01/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Task Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I happened to look at the README earlier and it still has old maven repository info.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
The github master has now the 2.0 info and we have a link for 1.4 to the communities site.




[JCBC-424] investigate adding a ping or other check to idle connections Created: 02/Mar/14  Updated: 20/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There have been a couple of reports of idle TCP connections causing operation failures. The underlying cause seems to be that some of these environments terminate seemingly idle TCP connections.

There are probably two solutions.
1) Don't let them go idle. On some existing thread or maybe with some kind of timer, do a version() operation or such regularly.
2) If it's possibly idle, send a ping before trying to send an op.

Of course, a 3rd solution is to automatically retry idempotent operations.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
I think we need to do this idle-ping style thing for another related bug edge case with CCCP, so I'll see if we can combine that for 1.4.2.
Comment by Mark Nunberg [ 02/Jun/14 ]
How about a simple ping() method? MySQL has it. We should have it in couchbase

btw ping won't help for those cccp edge cases either
Comment by Mark Nunberg [ 10/Jun/14 ]
I just realized this is attainable by something like the VERSION command :)
Comment by Michael Nitschinger [ 11/Jun/14 ]
Or even easier, what we do in 1.4.2 is a NOOP broadcast.




[JCBC-400] New feature test for credential encryption in cbc Created: 14/Jan/14  Updated: 07/Apr/14  Resolved: 14/Feb/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.0-dp1
Security Level: Public

Type: Task Priority: Major
Reporter: Deepti Dawar Assignee: Deepti Dawar
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
New feature test for credential encryption in cbc

 Comments   
Comment by Deepti Dawar [ 14/Jan/14 ]
http://review.couchbase.org/#/c/32355/




[JCBC-398] Extend CAS and asyncCAS operations to return new CAS value Created: 07/Jan/14  Updated: 12/May/14  Resolved: 12/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.2.3
Fix Version/s: 1.4.2
Security Level: Public

Type: New Feature Priority: Major
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The purpose being to prevent having to make a second get request if the CAS value is needed after the CAS operation.

 Comments   
Comment by loolek [ 15/Jan/14 ]
We need this feature too!
Comment by Michael Nitschinger [ 20/Feb/14 ]
Looking into it for 1.4, let's see.
Comment by Michael Nitschinger [ 12/May/14 ]
Hey Perry,

asyncCAS already supportes the new cas value (OperationFuture has a getCAS()) method, and we can't extend the cas() method to do it since its an ENUM and not a class.

If people really need it they, should use the asyncCAS method - makes sense?




[JCBC-396] Align getDesignDocument with other designDoc methods name-wise Created: 07/Jan/14  Updated: 09/Jan/14  Resolved: 09/Jan/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.3.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-397] Output Configuration Info on CouchbaseClient init Created: 07/Jan/14  Updated: 09/Jan/14  Resolved: 09/Jan/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.3.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-390] Refactor ClusterManager to use new HttpCore functionality. Created: 19/Dec/13  Updated: 31/Dec/13  Resolved: 31/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.3.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-388] Refactor View Connection Management Created: 16/Dec/13  Updated: 31/Dec/13  Resolved: 31/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core, Views
Affects Version/s: None
Fix Version/s: 1.3.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified