[JCBC-565] Adapt README with new location and getting started Created: 19/Sep/14  Updated: 19/Sep/14  Resolved: 19/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 2.0.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-564] Add release notes for 2.0 GA Created: 19/Sep/14  Updated: 19/Sep/14  Resolved: 19/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 2.0.0
Security Level: Public

Type: Task Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-563] View query will lock up after first attempt Created: 18/Sep/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 2.0-beta, 2.0-beta2
Fix Version/s: 2.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: casmeiron Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: deadlock, queryview
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: JAVA 1.8
MAC 10.9.4

Issue Links:
Dependency
depends on JVMCBC-40 View responses are stuck occasionally... Open

 Description   
If you try to run a simple query view twice at beta, you will notice that only #1 attempt will work, the second will wait indefinitely.

In beta-2, the query won't work at all, it will block right away!!

I've tested with async and sync mode in both releases.

Could you please take a look on that?
Thanks.


    @Test
    public void testAsync() throws Exception {
        CouchbaseCluster cluster = CouchbaseCluster.create(DefaultCouchbaseEnvironment.builder().build());
        Bucket bucket = cluster.openBucket("default");

        CountDownLatch latch = new CountDownLatch(10);

        for (int i = 0; i < 10; i++) {
            bucket
                    .async()
                    .query(ViewQuery.from("test", "test"))
                    .doOnNext(result -> {
                        if (!result.success()) {
                            System.err.println(result.error());
                        }
                    })
                    .flatMap(AsyncViewResult::rows)
                    .subscribe(row -> {
                        System.out.println(row.id());
                        latch.countDown();
                    });
        }

        latch.await();
    }




 Comments   
Comment by Matt Ingenthron [ 18/Sep/14 ]
Sergey: you may get to this before Michael
Comment by Michael Nitschinger [ 19/Sep/14 ]
Interesting, could you please share you view? When trying to reproduce this issue I found a different one, but it doesn't lock up immediately.

Thanks
Comment by Michael Nitschinger [ 19/Sep/14 ]
One thing, as a side note, I found in your view code is that you're counting down the latch for each row that returns, and that just once. You probably want more something like this?

                .flatMap(AsyncViewResult::rows)
                .doOnCompleted(latch::countDown)
                .subscribe(row -> System.out.println(row.id()));
Comment by Michael Nitschinger [ 19/Sep/14 ]
This appears to be a race condition if you actually do nothing with the results itself.

So if you do

while(true) {
      bucket.query(ViewQuery.from("beer", "brewery_beers")).toBlocking().single();
}

it will lock up, but if you add in a Thread.sleep(100) or do something with the results like printing it it will continue to work. Looking into it.
Comment by casmeiron [ 19/Sep/14 ]
Hi,
 The view emit all docs, exactly like the sample when you create a new view from console.
Thanks for improving my code with latch.
Comment by Michael Nitschinger [ 19/Sep/14 ]
I think the bug you were hitting is https://github.com/couchbase/couchbase-jvm-core/commit/805cb611deea9f36eb4cb6743733ff463e4b9f0c JVMCBC-39, but this uncovered another one which we're currently working on.
Comment by Michael Nitschinger [ 19/Sep/14 ]
One bug has been fixed, the other depends on JVMCBC-40. We are actively looking into fixing this for 2.0.1, for now please apply a lower timeout and retry the operation, this will help. 1.0.0 GA has an associated fix with a slightly different issue.
Comment by casmeiron [ 19/Sep/14 ]
Ok, I couldn't make it work setting the timeout for query and view. I will wait till you guys have a fix so we can try it again.
Thanks for the effort spent looking into this.
Best Regards.




[JCBC-562] Java 2.0 client feature test shows timeout exception on design document test Created: 17/Sep/14  Updated: 17/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to

 Description   
This is the only failed integration test from beta2

Class com.couchbase.client.java.DesignDocumentTest

java.lang.RuntimeException: java.util.concurrent.TimeoutException
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:481)
at rx.observables.BlockingObservable.single(BlockingObservable.java:348)
at com.couchbase.client.java.bucket.DefaultBucketManager.flush(DefaultBucketManager.java:138)
at com.couchbase.client.java.bucket.DefaultBucketManager.flush(DefaultBucketManager.java:60)
at com.couchbase.client.java.util.ClusterDependentTest.connect(ClusterDependentTest.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
at rx.internal.operators.OperatorTimeoutBase$TimeoutSubscriber.onTimeout(OperatorTimeoutBase.java:169)
at rx.internal.operators.OperatorTimeout$1$1.call(OperatorTimeout.java:42)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more




[JCBC-561] Rename binary to kv for less ambiguity Created: 16/Sep/14  Updated: 16/Sep/14  Resolved: 16/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-560] Add jar info in line with core io Created: 16/Sep/14  Updated: 16/Sep/14  Resolved: 16/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-559] Retry view responses if possible Created: 15/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
depends on JVMCBC-35 Expose View retry codes in response s... Open




[JCBC-558] Improve Javadoc for (Async)Bucket Created: 15/Sep/14  Updated: 19/Sep/14  Resolved: 19/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 2.0.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-557] Improve Javadoc for the .document package Created: 15/Sep/14  Updated: 19/Sep/14  Resolved: 19/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 2.0.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-556] Add support for optional decompression on legacy decodes Created: 15/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-555] Add support for JsonValue.NULL on the transcoders and documents Created: 15/Sep/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-554] Provide backoff handling utility classes for the user Created: 15/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.1.0
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-553] 2.0.0.beta hangs on 2nd query Created: 12/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Sean Colonello Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Client is Windows 8.1, server is CentOS 6 running 3.0 beta.

Attachments: Text File two-queries-hang.log    
Issue Links:
Duplicate
is duplicated by JVMCBC-28 ViewHandler can end up not completing... Resolved

 Description   
The following code hangs on the 2nd query.

CouchbaseCluster cluster = CouchbaseCluster.fromConnectionString("couchbase://192.168.1.75");
Bucket bucket = cluster.openBucket("trade-unittest", "blah").toBlocking().first();
bucket
.query(ViewQuery.from("devOnly", "allDocs"))
.flatMap(result -> result.rows())
.toBlocking()
.forEach(row -> logger.log(Level.INFO, row.id()));
bucket
.query(ViewQuery.from("devOnly", "allDocs"))
.flatMap(result -> result.rows())
.toBlocking()
.forEach(row -> logger.log(Level.INFO, row.id()));
cluster.disconnect();





[JCBC-552] Support other sources for key store Created: 12/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Cihan Biyikoglu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-551] Implement blocking API around async one Created: 11/Sep/14  Updated: 15/Sep/14  Resolved: 15/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Task Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-550] Java 2.0 SDK Backpressure Exception Created: 11/Sep/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This is a known issue and a fix is coming soon.
I work around the issue by waiting a bit longer between ops. It does reduces the number of backpressue exceptions.

Will run the test again after the fix is pushed.


 Comments   
Comment by Michael Nitschinger [ 19/Sep/14 ]
Moving this to 2.0.1 and non blocker because it seems to be a bug in the user code.
Comment by Wei-Li Liu [ 19/Sep/14 ]
@Michael backpressure exception decreased dramatically after latch being implemented.

I add 500 milliseconds timeouts for get, set and view ops, because I had some issues getting back the test result (get some timeout error)
https://github.com/couchbase/sdkd-java/commit/7f55481371c0e646b0f28fd7a41d01579c183b11

I was able to generate some test results, which contains high number of latency and error on timeout.
2.0 report
https://docs.google.com/spreadsheets/d/1WqmIXp6jzHIoWtsqmd15FmV7R-o0twDD-ZBBXevIK4w/edit#gid=1722831265

A simple passthrough test (no rebalance,restart...) is showing high latency as well
View : http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/passthrough-HYBRID/09-19-14/079432/b1de44abf73ddf45a636502e05fc2aaf-HT.html
MC: http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1209/passthrough-HYBRID/09-19-14/079432/b1de44abf73ddf45a636502e05fc2aaf-MC.html

I am running 10 threads and upsert 15000 docs. Do you have any suggestions here?




[JCBC-549] Implement setHashAlg() / Implement ability to change hashing algorithm Created: 11/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: New Feature Priority: Major
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt: start-finish
Relates to
relates to PYCBC-250 support for distributing data by hashkey Open

 Description   
CouchbaseConnectionFactoryBuilder.setHashAlg() is currently not operational.




[JCBC-548] user agent string is incorrect and doesn't work with some older HTTP core Created: 10/Sep/14  Updated: 10/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I found that the current client can't be used in CDH5 because a JavaLangNoSuchMethodError will be thrown with org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V since the older 4.2 Apache httpcore does not have that method of defining a string and it takes precedence in the classpath.

In the process, I found that this header wasn't being set correctly anyway.

Filing this issue to remove the header. If we get a chance, it should be fixed by re-adding with reflection or whatever method 4.2 provides.




[JCBC-547] Paginator hasNext() works incorrectly on small results Created: 09/Sep/14  Updated: 09/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: raycardillo Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: javaclient
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
While working with the Paginator in our QA system I traced down a bug that will only happen when the page size is greater than the number of items in the underlying view/query. We wanted to call hasNext() multiple times because one layer of code validates that it has a good Paginator (due to other problems we were having with our cluster). However, because the actual response is fetched during hasNext(), and because the FINISHED state is set as a result of that, only the first hasNext() will return true even though we have not retrieved the response with next() yet.

For example, we should be able to call, hasNext() multiple times, and then state is only changed to indicate FINISHED after we actually retrieve the response using next().

After tracing this bug down, I can see that hasNext() returns false right away if FINISHED is true, and I can see that the FINISHED state is set during the fetch that happens during hasNext(). So when there is less than one page of data available, the first hasNext() will return true, and the next hasNext() will return false, which is a problem if the consumer has not actually called next() to fetch the response.

I think the simple fix here is to only set the FINISHED state in next() so you know that the data has actually been retrieved by the consumer. That way the consumer is free to ask hasNext() as many times as required for their logic, and it will not effect internal state until next() is called.




[JCBC-546] Expected Behavior of FailureMode.Cancel Created: 08/Sep/14  Updated: 16/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Matt Ingenthron
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
Customer has questions on the behavior of FailureMode.Cancel:

1) Is TimeoutExceptionThreshold irrelevant when using FailureMode.Redistribute ? The node is not deemed down even when the number of consecutive failures exceed the threshold.
2) In what scenarios is it advisable to use Redistribute ?
3) Whats the .NET equivalent.




[JCBC-545] Couchbase Connection leaks to a single node in the cluster Created: 05/Sep/14  Updated: 15/Sep/14  Due: 14/Sep/14  Resolved: 15/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.2
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Critical
Reporter: Justin Michaels Assignee: Michael Nitschinger
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We are observing 2x the number of connections open on a single node in a given Couchbase cluster. All connections are on port 11210.

The development team believes they have identified a bug that creates this behavor during a cluster change and a new map being distributed to the Couchbase clients. Below is their observation.
... there in the list to randomize the nodes, but there’s a bug in the constructor of Couchbase SDK class com.couchbase.client.vbucket.provider.BucketConfigurationProvider

Here’s the code:

  public BucketConfigurationProvider(final List<URI> seedNodes,
    final String bucket, final String password,
    final CouchbaseConnectionFactory connectionFactory) {
    config = new AtomicReference<Bucket>();
    configurationParser = new ConfigurationParserJSON();
    httpProvider = new AtomicReference<ConfigurationProviderHTTP>(
      new ConfigurationProviderHTTP(seedNodes, bucket, password)
    );
    refreshingHttp = new AtomicBoolean(false);
    pollingBinary = new AtomicBoolean(false);
    observers = Collections.synchronizedList(new ArrayList<Reconfigurable>());
    binaryConnection = new AtomicReference<CouchbaseConnection>();

    this.seedNodes = Collections.synchronizedList(new ArrayList<URI>(seedNodes));
    this.bucket = bucket;
    this.password = password;
    this.connectionFactory = connectionFactory;
    potentiallyRandomizeNodeList(seedNodes);

    disableCarrierBootstrap = Boolean.parseBoolean(
      CouchbaseProperties.getProperty("disableCarrierBootstrap", "false"));
    disableHttpBootstrap = Boolean.parseBoolean(
      CouchbaseProperties.getProperty("disableHttpBootstrap", "false"));
  }


The potentiallyRandomizeNodeList method will randomize the list passed to it and in fact, in our clients we ARE randomizing the two-element list we are passing.
Unfortunately, the wrong list is being passed to this method. The code should be randomizing the list stored as a member variable.
The line should read
potentiallyRandomizeNodeList(this.seedNodes);

because four lines above a completely new list is constructed by making a shallow copy of the input argument, seedNodes.

The list later used to get the configuration information is always used ordered and that’s why all our management connections are to the same node, the first one in our list.

I found this in v1.4.2 of the client, but v1.4.4 seems to have the same code so I don’t think it’s been fixed yet.

 Comments   
Comment by Dan Douglas [ 06/Sep/14 ]
I found this issue.

The symptom is that the configuration information is always obtained from the first node in seed list passed and the shuffle is not done.

The line that does the shuffle in the constructor needs to reference the seedNodes member variable, not the argument. Needs to add "this." like so

potentiallyRandomizeNodeList(this.seedNodes);






[JCBC-544] Does not throw error when querying a non existent view/designdoc Created: 05/Sep/14  Updated: 16/Sep/14  Resolved: 16/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Bug Priority: Major
Reporter: Subhashni Balakrishnan Assignee: Sergey Avseyev
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Here is the code I used. This doesn't call onError.
 

ViewQuery queryTemplate = ViewQuery.from("test","dummy");
    queryTemplate.stale(Stale.FALSE);
    Observable<ViewRow> viewRowObservable = bucket.query(queryTemplate).flatMap(new Func1<ViewResult, Observable<ViewRow>>() {
      public Observable<ViewRow> call(ViewResult res) {
        return res.rows();
      }
    });

   viewRowObservable.subscribe(new Observer<ViewRow>() {
     @Override
     public void onCompleted() {
      System.out.println("Completed");
     }

     @Override
     public void onError(Throwable throwable) {
      System.out.println(throwable.getMessage());
     }

     @Override
     public void onNext(ViewRow viewRow) {
     }
   });

 Comments   
Comment by Matt Ingenthron [ 09/Sep/14 ]
Sergey: Can you see if this is reproducible off the current master and if so investigate?
Comment by Michael Nitschinger [ 09/Sep/14 ]
Hi,

some remarks: it's just not implemented. The core should send the response back as a failure (see the status). Chain in an onNext and check for the error. if it is, throw a ViewDoesNotExist execption. Put the exception in the error namespace, the others are also in there.
Comment by Sergey Avseyev [ 12/Sep/14 ]
http://review.couchbase.org/41369
Comment by Michael Nitschinger [ 16/Sep/14 ]
I've removed the blocker, because right now you can get access to the error on the observable.

We need to work through some more tricky semantics, but this will not break the API at this point. Sergey will update this ticket once there is more progress, but its not holding us back for the beta2.
Comment by Michael Nitschinger [ 16/Sep/14 ]
Okay one step back, we'll add an exception for this in beta2 (ViewDoesNotExistException) if 404 is returned. For more tricky cases we'll handle it in separate tickets.




[JCBC-543] IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest Created: 05/Sep/14  Updated: 05/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core, Infrastructure
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This could be user error, but when trying the couchbae hellworld tutorial: http://docs.couchbase.com/prebuilt/java-sdk-2.0-beta/topics/hello-couchbase.html, I receive the following exception:

Sep 05, 2014 10:30:02 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Connected to Node 127.0.0.1
Sep 05, 2014 10:30:07 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Disconnected from Node 127.0.0.1
Sep 05, 2014 10:30:07 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Connected to Node jmorris-pc
Sep 05, 2014 10:30:07 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 2 com.couchbase.client.core.RequestEvent@7bcfceb7
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.core.node.locate.BinaryLocator.locateForCouchbaseBucket(BinaryLocator.java:126)
at com.couchbase.client.core.node.locate.BinaryLocator.locate(BinaryLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Exception in thread "pool-1-thread-2" java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.core.node.locate.BinaryLocator.locateForCouchbaseBucket(BinaryLocator.java:126)
at com.couchbase.client.core.node.locate.BinaryLocator.locate(BinaryLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
... 3 more

Additional logging included. I am connecting to localhost CB server 2.5.




[JCBC-542] Handle TMPFAIL and expose it Created: 04/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-541] incr() returns -1 Created: 02/Sep/14  Updated: 09/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: RICHARDS PETER Assignee: Matt Ingenthron
Resolution: Unresolved Votes: 0
Labels: info-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Hi,

I am using incr() of couchbase client to increase a counter key in a bucket. I found this method very useful in some of our usecases. Around 150 threads from four machines increments current value of counter by 10 and use the values for some internal logic. Initially everything worked as expected. However a few days back I realized that couchbase returns -1 for some of the incr() method calls. That is how I noticed that incr() in MemcachedClient could return -1 if there is a failure to update the counter:
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.3/net/spy/memcached/MemcachedClient.html

I have a few queries about this return value:
1. Could you please tell me the cases in which incr() would return -1? I identified one case. If the RAM quota allocated to the bucket is exhausted, we are likely to get -1. During this time addition of key-value pairs into the same bucket using add() will return false through the OperationFuture object. What are the other cases that would return -1 in the incr()?

2. Could you please tell me how the following method call works?
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.3/net/spy/memcached/MemcachedClient.html#incr%28java.lang.String,%20long,%20long,%20int%29

I would like to know when the default value would be returned and when -1 would be returned. I think the default value would be returned if it is possible to insert the value into couchbase and -1 would be returned if it is not possible to retrieve the counter and update it. Am I correct?

3. Till now my project did not have a use case to increment negative values. However if somebody wants to increment -2 by 1 then the result would be -1. How will the client call be able to distinguish failure and actual increment in this case? Wouldn't it be better to throw a checked exception from incr() than returning -1? I think with checked exception we can provide more information about the cause of failure.

Richards Peter.




[JCBC-540] Enable SDK 1.x & 2.x to be deployed side by side in a single JVM. Consider renaming couchbase-client to java-client Created: 01/Sep/14  Updated: 09/Sep/14  Resolved: 09/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Dan Douglas [ 02/Sep/14 ]
Thanks for considering this.

Renaming the artifact is a start. However, the package names should be different, too, to make sure that the class namespace has no clashes.

The goal should be coexistence of both 1.4 SDK and 2.0 SDK clients in the same JVM.

Motivation is Couchbase use by large, distributed teams that will not all migrate in lockstep, yet contribute code that runs in the same JVM.

Lack of separation at both Maven artifact AND class namespace level will impede migration to CB Java 2.0 SDK.
Comment by Michael Nitschinger [ 02/Sep/14 ]
Hm, they both have com.couchbase.client in common, but the new one is now under com.couchbase.client.java and the core is com.couchbase.client.core

Is that sufficient or would the new one be completely different and not under com.couchbase.client at all?
Comment by Michael Nitschinger [ 02/Sep/14 ]
I just tried adding both as dependencies (I locally renamed the new one to java-client and pushed a new artifact) and it seems to work nicely:

    public static void main(String... args) throws Exception {

        CouchbaseClient client = new CouchbaseClient(
            Arrays.asList(URI.create("http://127.0.0.1:8091/pools")),
            "default",
            ""
        );

        Cluster cluster = CouchbaseCluster.create();
        Bucket bucket = cluster.openBucket().toBlocking().single();

        client.set("id", "value1234").get();
        System.out.println(bucket.get("id", LegacyDocument.class).toBlocking().single());

    }
Comment by Dan Douglas [ 02/Sep/14 ]
Doesn't have to be completely different.... just as long as there are no clashes in ANY of the class names.

Different prefixes are ok.

So as long as the SDK 2.0 packages always start with

com.couchbase.client.java
com.couchbase.client.core
com.couchbase.client.deps

and the SDK 1.4 maintenance never adds those paractular packages, it should be ok.

So far, the 1.4 version uses these packages...

com.couchbase.client
com.couchbase.client.clustermanager
com.couchbase.client.http
com.couchbase.client.internal
com.couchbase.client.protocol
com.couchbase.client.vbucket

It seems a little ripe for error, however. Should someone maintaining 1.4 ever forget and slip in .java, .core or .deps ... but like you said, so far so good.

Taking a step back, though, since the 2.0 SDK doesn't actaully have a "client" class (uses Cluster and Bucket objects instead) and the use of "java" seems somewhat superfluous here, perhaps you'd consider dropping "java" and changing the 2.0 SDK packages to

com.couchbase.sdk

or

com.couchbase.jsdk

or

com.couchbase.jcbc

or

something else....



Just a thought. What you have works, although it seems a little subtle and open to creating confusion.

There's precedent for changing package/artifact names together, for example Apache Commons Lang 3 (http://commons.apache.org/proper/commons-lang/javadocs/api-3.3.2/)
Comment by Cihan Biyikoglu [ 02/Sep/14 ]
added the goal to the title for clarity: side by side deployment of 1.x and 2.x SDKs in a JVM.
Comment by Michael Nitschinger [ 04/Sep/14 ]
I'm actually very open to rename the artifact to java-client (http://review.couchbase.org/#/c/41194/), but I'm not sure its a good idea to rename the namespace. Btw "java" is in there to align with "core" for "core-io" and also in the future for other language bindings like "scala" and so forth.

Since the 1.* branch is mostly in bugfix mode right now, I don't think there will be much room for conflict, since all classes added by the new one are under one level deeper and the only thing we need to make sure is not add a "java" or "core" namespace in the current stable SDK.

Would that make sense?
Comment by Dan Douglas [ 04/Sep/14 ]
Michael, you're the man. If you think it's going to be ok to just rename the artifact, then I'll agree with you. If someone introduces a clash later in a "fix", it'll just be a bad fix that will need to be redone. Perhaps you can introduce something in your static analysis or automated test suite to catch that.
Comment by Michael Nitschinger [ 09/Sep/14 ]
Merged!




[JCBC-539] Consider a way to expose target key nodes for user-level circuit breakers Created: 01/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File HelloWorld.java     Java Source File NodeLocator.java    
Issue Links:
Relates to
relates to CCBC-505 Allow user-side modification of node'... Open

 Comments   
Comment by Dan Douglas [ 02/Sep/14 ]
Attached hacked/reflected workaround I used to get locator information and simple HelloWorld showing usage.

As indicated, real usage is for creating Hystrix command wrappers around CB functions to do circuit breakers for resiliancy in the face of failing (but not yet failed over) nodes.
Comment by Matt Ingenthron [ 02/Sep/14 ]
Thanks Dan! This is something we've been discussing a bit and I really wonder if in general we want a "fast-fail" behavior to be defined. Maybe we need another word. Basic idea is that there are some use cases where people don't want to wait for the timeout value while the connection is known to be down. They want to fast-fail while that connection is attempted to be rebuilt. I didn't read all the code, but is this the kind of behavior you were going for by checking node status?
Comment by Dan Douglas [ 03/Sep/14 ]
Fail fast is one aspect.

However, our plan also calls for a policy driven fallback

- readFromReplica
- read/write to secondary cluster
- user supplied fallback

What I want is access to the locator data so I can associate a key with a node (both primary and replicas).

I don't intend to check the client's status of the node through any CB API. I'll identify the iffy nodes using the outcome of operations associated with them by wrapping them in Hystrix commands. The locator info is just to creat the name of the command.

For even more context, here’s a small deck describing the work I’m doing to add more resiliency to the DUKES wrapper we use for configuration and application monitoring of Couchbase. It includes a short six-minute embedded video demonstrating how the proposed implementation shields applications from badness when a CB node is failing, but has not yet failed over.
https://www.dropbox.com/s/8hue2za1kjje7zu/DUKESv2-Resiliant-Hystrix-Commands.pptx?dl=0 (156 MB)

Some folks have said that they’ve had trouble running the embedded video from the PPT. Here’s a direct link to the .mp4
https://www.dropbox.com/s/zaek5042bk3rhl0/DukesV2Demo.mp4?dl=0 (153 MB)

The NodeLocator implementation attached is what I’ll need to do in order to implement DUKESv2 onto of Couchbase SDK 2.0. I’m not really looking forward to maintaining the hack. I’m hoping to convince you to add back some sort of locator information to the SDK so that I won’t need to.
Comment by Mark Nunberg [ 03/Sep/14 ]
If I may chime in here a bit. The same feature has been requested for the C client library. I'd also say that rather than stuffing all the logic into the client library, providing the facilities and tools for a user to manage such a "circuit breaker" functionality.




[JCBC-538] Consider adding bulk APIs Created: 01/Sep/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.1.0
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-537] Setting Reader/Writer Worker value on Server causes ParseException on Client Created: 29/Aug/14  Updated: 09/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Minor
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File output.log    
Issue Links:
Dependency
depends on MB-12104 Carrier Config missing after R/W Conc... Open
Duplicate

 Description   
When setting the "Number of suggested reader/writer workers" in the Admin UI to 8, the 1.4.4 client then begins to see ParseException errors during the bootstrap phase. The client appears to continue as expected after that, but the error is concerning. The exception does not occur in the .NET or C client libraries.

To repro:

* Set the worker value to 8 in the UI
* Run the HelloWorld example from:

http://www.couchbase.com/communities/java/getting-started

Relevant code is:

// (Subset) of nodes in the cluster to establish a connection
    List<URI> hosts = Arrays.asList(
      new URI("http://127.0.0.1:8091/pools")
    );
 
    // Name of the Bucket to connect to
    String bucket = "default";
 
    // Password of the bucket (empty) string if none
    String password = "";
 
    // Connect to the Cluster
    CouchbaseClient client = new CouchbaseClient(hosts, bucket, password); // <- exception here

When running the example, the response is consistently:

2014-08-29 15:36:21.979 INFO net.spy.memcached.auth.AuthThread: Authenticated to /<ip_address>:11210
2014-08-29 15:36:22.014 WARN com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could not parse config, retrying bootstrap.
java.text.ParseException: A JSONObject text must begin with '{' at character 0 of
at com.couchbase.client.vbucket.config.ConfigurationParserJSON.parseBucket(ConfigurationParserJSON.java:148)



 Comments   
Comment by Matt Ingenthron [ 29/Aug/14 ]
If you crank up the log level, it should print out what the cluster is returning. Maybe the .NET and C clients are probably seeing the same behavior but not logging the warning?
Comment by Jeff Dillon [ 29/Aug/14 ]
In .NET I used the following log settings, and didn't see the error:

      <priority value="ALL" />
      <level value="DEBUG" />
Comment by Matt Ingenthron [ 29/Aug/14 ]
Yep, might be a bug that .NET doesn't log a warning on bad response and goes on to the next node.

The parse error comes from Jettison. https://github.com/couchbase/couchbase-java-client/blob/669c9580674082b2acccd769d9da5e1b3bb3e417/src/main/java/com/couchbase/client/vbucket/config/ConfigurationParserJSON.java#L148

It'd be really helpful if you could get the log to tell us what the cluster is returning.
Comment by Jeff Dillon [ 29/Aug/14 ]
Attaching Java log using:

logging.properties
handlers = java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=output.log
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
com.couchbase.client.vbucket.level = FINEST
com.couchbase.client.vbucket.config.level = FINEST
com.couchbase.client.level = FINEST
Comment by Michael Nitschinger [ 01/Sep/14 ]
Interesting, it could also be causing a latent parsing bug in the handler which has not been uncovered yet. I'll try to reproduce it.
Comment by Michael Nitschinger [ 01/Sep/14 ]
Wow, this looks like a server issue!

When you change the number of r/w threads the binary get config for CCCP returns an empty response.
Comment by Michael Nitschinger [ 01/Sep/14 ]
Please see http://www.couchbase.com/issues/browse/MB-12104 for more info
Comment by Michael Nitschinger [ 01/Sep/14 ]
While this is a server bug, I think we can do better on the binary bootstrap in the SDK to not try to apply a config like this and keep moving on, falling back to http.




[JCBC-536] Interpretation and Performance Impact of Client Profiling Created: 29/Aug/14  Updated: 17/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to

 Description   
Customer is using client side profiling as suggested in:

http://blog.couchbase.com/whats-new-couchbase-java-sdk-12

Do we have documentation on the Meters, Counters and Histogram output? What would the performance impact be, if any? Can this be safely left in place in a Production environment, for example?

 Comments   
Comment by Jeff Dillon [ 02/Sep/14 ]
Here are specific asks:

1) We need documentation on counters, meters and histogram output.
2) Can we run clients with MetricType = PERFORMANCE for extended duration? What's the overhead?
3) Can we change the net.spy.metrics.reporter.type to hook it up to Graphite? The metrics-core library does offer support for a GraphiteReporter. Reference: http://metrics.dropwizard.io/manual/graphite/#manual-graphite
4) Is it possible to add custom metrics over and above what's being gathered by the Couchbase java client? If so, how?
Comment by Deepu Bhatia [ 09/Sep/14 ]
Hi Michael, Matt,
Can we please get an update on this?

Thanks,
-Deepu
Comment by Michael Nitschinger [ 10/Sep/14 ]
Hi,

sorry, I've been quite busy lately...

Currently there is not much more documentation available, and metrics can be used in production environments. That said, while the metrics library is designed to be used in production it is not "zero overhead" and should be used with care. PERFORMANCE just logs less information, only those needed to find performance infos.

wrt to graphite: yes this should be possible to do, but I've never tried graphite on my own together with metrics.

You can use the MetricCollector provided on the Factory and use the methods like add counter with a name and then you can increment the counter as well. This API is used in the core, but to my knowledge is not (widely) used out there in general, but it should definitely work. If it does not, its a bug.
Comment by Deepu Bhatia [ 11/Sep/14 ]
Hi Michael,
At the bare minimum, I'd at least like to get the definitions for each of the metrics so that there's no misunderstanding. Here's a list of files being created when we use CSV logging.

[MEM] Shutting Down Nodes (NodesToShutdown).csv
[MEM] Response Rate: Success.csv
[MEM] Response Rate: Retry.csv
[MEM] Response Rate: Failure.csv
[MEM] Response Rate: All (Failure + Success + Retry).csv
[MEM] Request Rate: All.csv
[MEM] Reconnecting Nodes (ReconnectQueue).csv
[MEM] Average Time on wire for operations (µs).csv
[MEM] Average Bytes written to OS per write.csv
[MEM] Average Bytes read from OS per read.csv

This is critical to understand to unblock us. I understand that it's not zero overhead. We'll benchmark the performance with PERFORMANCE mode enabled and decide accordingly.
Comment by Deepu Bhatia [ 12/Sep/14 ]
Michael, Matt,
This has become critical for us. We have client side profiling capability now but we don't have an exact definition of the metric. This is a show stopper for us.
Comment by Matt Ingenthron [ 13/Sep/14 ]
Depu: The ones mentioned as "rate" are counters, so you'll need to collect with regularity. The average time and byte counts are just that.

Is there a particular question you're trying to answer? Maybe with that we can give some more specific guidance. I understand it may have to do with latency and where it is? Does the average time on wire correlate to what you're seeing from cbstats timings would be the thing I would be checking for. I'd also be looking for reconnect to be non-zero and retry rate being non-zero.
Comment by Deepu Bhatia [ 16/Sep/14 ]
Hi Matt,
Here's what I am specifically looking for:

[MEM] Shutting Down Nodes (NodesToShutdown).csv - what does this mean?
[MEM] Response Rate: Success.csv - what does this mean?
[MEM] Response Rate: Retry.csv - What retries are this?
[MEM] Response Rate: Failure.csv - What failures does this cover? Is this tracking timeout exceptions?
[MEM] Response Rate: All (Failure + Success + Retry).csv
[MEM] Request Rate: All.csv
[MEM] Reconnecting Nodes (ReconnectQueue).cvs - What is this metric tracking?
[MEM] Average Time on wire for operations (µs).csv - Is this pure network time? Is it in microseconds?
[MEM] Average Bytes written to OS per write.csv
[MEM] Average Bytes read from OS per read.csv

Comment by Michael Nitschinger [ 17/Sep/14 ]
Hi Deepu,

here is more info on each specific metric:

[MEM] Shutting Down Nodes (NodesToShutdown).csv - what does this mean?

>> contains nodes which are currently pending shutdown in the IO client. This means that if a node is failovered or removed, at some point it needs to be shutdown. This mostly indicates that a cluster is currently undergoing some changes.

[MEM] Response Rate: Success.csv - what does this mean?

>> How many responses returned with a "success" state from the server. so no failures or retries.

[MEM] Response Rate: Retry.csv - What retries are this?

>> how many responses returned from the server which are to be retried. This is mostly covered by "not my vbucket" responses that are coming up during a rebalance operation. If such ops are to be seen, then the cluster is currently or recently undergoing a rebalance operation.

[MEM] Response Rate: Failure.csv - What failures does this cover? Is this tracking timeout exceptions?

>> Operations that had been failed from the server side. This should normally not be larger than 0, if so something is wrong. This can indicate all forms of errors and the logs need to be investigated further.

[MEM] Response Rate: All (Failure + Success + Retry).csv

>> the response rate for all those responses combined (to get an aggregated view on the response rate)

[MEM] Request Rate: All.csv

>> rate of outgoing requests

[MEM] Reconnecting Nodes (ReconnectQueue).cvs - What is this metric tracking?

>> the number of all nodes that are currently waiting to be reconnected. this could be due to too many failing/timing out ops (issuing a manual reconnect), or the socket is closed upon us. A reconnecting node is not able to process operations, so this is indicating some form of unstable/fluent state.

[MEM] Average Time on wire for operations (µs).csv - Is this pure network time? Is it in microseconds?

>> this is nearly network time, and yes in microseconds. It is measured once we put the op over to the JVM and the second time when we start reading it off the JVM NIO parts. So it does not include only networking time, but also OS and JVM socket processing. But it cuts out other parts like transcoding.

[MEM] Average Bytes written to OS per write.csv

>> this is an indication of how well the syscalls are used, the higher the batch size the better the utilization. Lower utilization means more overhead for packets and less efficient network handling.

[MEM] Average Bytes read from OS per read.csv

>> see above, just for reads.




[JCBC-535] Add documentation on different document types Created: 28/Aug/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-534] Enhance documentation how JSON is handled Created: 28/Aug/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-533] Add documentation for 1.x migration Created: 28/Aug/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Matt Ingenthron [ 13/Sep/14 ]
disregard my earlier comment on this sergey.




[JCBC-532] Implement Common Flags Created: 27/Aug/14  Updated: 11/Sep/14  Resolved: 11/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Task Priority: Major
Reporter: Brett Lawson Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Implement the Common Flags formatting as defined by the specification.

 Comments   
Comment by Michael Nitschinger [ 11/Sep/14 ]
dups 525




[JCBC-531] Expose Diagnostics and debug dump them on startup Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-530] Exception when using java client both in Spark and MapReduce Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sun Chen Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: Hadoop,Dependencies,HttpCore, Spark,
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h
Environment: Haddop2.4.1,Spark0.13,Java1.7,Ubuntu12.04


 Description   
I am using Couchbase java client(version 1.4.4),I writed some simple mapreaduce job and spark job to write and read data from Couchbase.But I always got an exception.Here is the exception information

-------------------------------------------------------------------------------------------------------------------------------------------------
2014-08-25 13:12:56,048 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V
at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156)
at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:31)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:20)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
-------------------------------------------------------------------------------------------------------------------------------------------

14/08/09 10:19:32 WARN Utils: Your hostname, precise64 resolves to a loopback address: 127.0.1.1; using 192.168.33.10 instead (on interface eth1) 14/08/09 10:19:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/08/09 10:19:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-09 10:19:40.569 INFO net.spy.memcached.auth.AuthThread: Authenticated to /127.0.0.1:11210 2014-08-09 10:19:40.677 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication. 2014-08-09 10:19:40.693 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=localhost/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue 2014-08-09 10:19:40.702 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{bucket='Result', nodes=[http://127.0.0.1:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500} Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156) at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194) at com.eyespage.dc.etl.c2s.Run$.main(main.scala:16) at com.eyespage.dc.etl.c2s.Run.main(main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2014-08-09 10:19:40.836 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210 - See more at: http://www.couchbase.com/communities/q-and-a/exception-when-using-scala-call-java-sdk%EF%BC%8Cand-run-spark#sthash.CmL5dcRz.dpuf








[JCBC-529] Add support for spatial Created: 22/Aug/14  Updated: 11/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-528] Add shortcut JsonObject and JsonArray factories Created: 22/Aug/14  Updated: 04/Sep/14  Resolved: 04/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-527] Add cluster info Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-526] Add support for bucket management operations Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-525] Add support for full JSON compat docs & common flags Created: 22/Aug/14  Updated: 16/Sep/14  Resolved: 15/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-524] Add support for custom transcoders Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-523] Add support for memcached buckets Created: 22/Aug/14  Updated: 15/Sep/14  Resolved: 15/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Sergey Avseyev
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Matt Ingenthron [ 13/Sep/14 ]
Sergey: I believe this is done. Can you verify?
Comment by Sergey Avseyev [ 15/Sep/14 ]
Verified, memcache buckets supported with exception of APIs which are not available there (like locking)




[JCBC-522] Implement bucket info. Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-521] VBucketNodeLocator.java getPrimary() and getReplica() log messages are homologous Created: 21/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
VBucketNodeLocator.java has these two methods:
- getPrimary(String k)
- getReplica(String key, int index)

These two write almost the same warning log message when it can't access a node which holds the key.

getPrimary() logs: "for which no server is responsible in the cluster map (-1)."
while
getReplica() logs: "for which no server is responsible in the cluster map."
(without -1 in the message)

 Comments   
Comment by Koji Kawamura [ 21/Aug/14 ]
These two methods log almost the same warning messages, but since those two are indicating different level of severity, those better to be differentiated.
Comment by Michael Nitschinger [ 25/Aug/14 ]
I think it would be okay to push the replica one down to INFO level.Would that be sufficient? I'd rather not change the message itself in a bugfix release.
Comment by Koji Kawamura [ 25/Aug/14 ]
Changing log level of missing replica node to INFO sounds reasonable. Thank you!




[JCBC-520] Running SSL Test on 2.0 Client hangs when certificate is not valid Created: 20/Aug/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: 2.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The SSL test I make can be seen from
https://github.com/weilliu/couchbase-java-client/commit/97c3144692627d6e657a0cc157f5f9aa644de0f2
http://review.couchbase.org/#/c/40763/

Currently if we enable ssl, and when it hits a certificate which is not valid/not exist, the test would hang


11:38:16.868 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STARTED
11:38:17.843 [DEBUG] [TestEventLogger]
11:38:17.843 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STANDARD_OUT
11:38:17.843 [DEBUG] [TestEventLogger] File copied!
11:38:17.844 [DEBUG] [TestEventLogger] Reconnect with the Server
> 11:39:06.2 [ DEB] [org.gradle.process.internal.ExecHandleRunner] Abort requested. Destroying process: Gradle Worker 1.
> Building 85% > :integrationTest > 29 tests completed, 29 skippedWei-Lis-MacBook-Pro:couchbase-java-client wei-li$






[JCBC-519] Timeout Not Honored on First Call to Java asyncGetBulk Created: 11/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File App.java    
Issue Links:
Duplicate

 Description   
This was observed on Couchbase Server 2.2.0. The same behavior is not evident on Couchbase Server 2.5.1. It appears to be independent of any Java client version. The very first call of asyncGetBulk does not respect the timeout value, and continues to process and wait. Subsequent calls do properly respect the timeout value. One interesting note is that this only occurs when the timeout value is "close" to the actual execution time. That is, a very short timeout is indeed honored.

Steps to reproduce:

* Download a Java client, such as http://packages.couchbase.com/clients/java/1.4.3/Couchbase-Java-Client-1.4.3.zip

* Unzip

* Copy the attached App.java into the unzipped directory

* Compile and run:

javac -cp couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar App.java

java -cp .:couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar:jettison-1.1.jar:netty-3.5.5.Final.jar:commons-codec-1.5.jar:httpcore-4.3.jar:httpcore-nio-4.3.jar App > 143.txt

* Note the following output in the first group. It would be expected that the first call would timeout, but does not:

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=108ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=16ms


Exception:********************
Timeout future is: true
Operation timed out. - failing nodes: 10.4.2.107/10.4.2.107:11210, cbhost.domain/10.4.2.57:11210
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms


ASK - Determine why the first call does not honor the timeout, and ideally suggest a workaround for Couchbase Server 2.2.0, short of upgrading to Couchbase Server 2.5.1 (where the issue does not appear exist, and therefore looks to be a server-side issue)




 Comments   
Comment by Aleksey Kondratenko [ 11/Aug/14 ]
Not honoring timeouts cannot be server side issue. Passing to Michael who owns couchbase java sdk
Comment by Jeff Dillon [ 12/Aug/14 ]
Please disregard my previous comments with respect to being apparently fixed with Couchbase Server 2.5.1. This behavior evidences itself across all CB server versions, and also reproduces with CB Java client 1.4.4. In my latest 2.5.1 test, I set the timeout to 30 ms, and see the same incorrect behavior. So this is indeed a client issue (as noted), apologies on the initial misinformation.
Comment by Michael Nitschinger [ 14/Aug/14 ]
A few remarks:

- The simple benchmark also profiles logging and so forth.
- System.currentTimeMillis() precision is sometimes beyond all joke, always use nanotime.

So something like this is more accurate:

        while (low == 10) {
            try {
                startTime = System.nanoTime();
                future = client.asyncGetBulk(Main.getKeys());
                future.get(timeout, TimeUnit.MILLISECONDS);
                endTime = System.nanoTime();
                System.out.println("Timeout future is: " + future.isTimeout());
                System.out.println("Fetched " + Main.getKeys().size() + " keys");
            } catch (Exception e) {
                System.out.println("\nException:********************\nTimeout future is: " + future.isTimeout());
                System.out.println(e.getMessage());
            }
            operationTime = endTime - startTime;
            System.out.println("Timeout set to:" + timeout + "\ngetMulti : Time taken=" + TimeUnit.NANOSECONDS.toMillis(operationTime) + "ms\n");
        }

Now also note that profiling on the blocking .get() call not only waits on the latch where the timeout is applied, but it also puts all data together and so forth. So as far as I can see it the timeout is honored, its just not what you benchmark that close. Keep in mind that the first few calls are subject t JVM warmup, optimizations and so forth which can take some time. It's not a huge surprise that the first few calls are slightly slower, but I guess thats not what the JVM optimizes for compared to a long-running server.

Does that help?
Comment by Jeff Dillon [ 14/Aug/14 ]
I am seeing the same behavior when using nanoTime. And it's not slightly slower, it's an order of magnitude difference. And I don't think it's JVM warmup either, as different timeout values show consistent differences every time. Using a short time out value works as expected, but is not the timeout value they desire. When the timeout is "close" to the actual execution time, it experiences this behavior. This is what I'm seeing in my tests, and is what the customer is seeing also.
Comment by Matt Ingenthron [ 20/Aug/14 ]
Jeff Dillon: I understand from James this has been observed on 2.5.1 now as well. Can you confirm and update it appropriately? Also, I've moved it to JCBC, since the investigation has lead here.
Comment by Jeff Dillon [ 20/Aug/14 ]
Yes, please see my previous comment from 12/Aug/14 1:47 PM




[JCBC-518] no intro page on the javadoc Created: 18/Aug/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: 2.0.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Goal is to have the index page for the javadoc give an introductory overview so a developer knows which class to begin with. The 1.4 javadoc does this already, but the 2.0 doesn't have introductory HTML getting built in to the project. At a minimum, it should describe accessing a cluster and a bucket with a code sample that can be cut and paste into a working program.

 Comments   
Comment by Matt Ingenthron [ 13/Sep/14 ]
Sergey: This should be an easy one. There is a similar change on the release14 branch.




[JCBC-517] Add overloads for Persist/Replicate only Created: 18/Aug/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-516] All internal code should use the provided scheduler from the environment Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Michael Nitschinger [ 21/Aug/14 ]
fixed in the core.




[JCBC-515] Simplify environment and collapse properties Created: 18/Aug/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-514] Only shutdown environment if not shared Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-513] Implement bucket.close() Created: 18/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-512] $HOST in Refresher against localhost Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
carrier (and maybe also http refresher) don't replace $HOST

 Comments   
Comment by Michael Nitschinger [ 21/Aug/14 ]
Done in core.




[JCBC-510] Allow optional non-persistent view connections Created: 18/Aug/14  Updated: 18/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-509] Client doesn't shut down Created: 17/Aug/14  Updated: 27/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Adam Honen Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Java 87 update 11.
Windows 7 Service Pack 1 & Linux.

Attachments: Text File disconnection.log    
Issue Links:
Dependency

 Description   
Despite calling cluster.disconnect(), there are remaining non daemon (netty) threads which prevent the JVM from shutting down.

A thread dump from my machine when the program should have terminated:

Full thread dump Java HotSpot(TM) Client VM (25.11-b03 mixed mode):

"DestroyJavaVM" #22 prio=5 os_prio=0 tid=0x0016d800 nid=0x1f38 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"RxComputationThreadPool-4" #21 daemon prio=5 os_prio=0 tid=0x15255400 nid=0x17ac waiting on condition [0x18c7f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a02bd00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-3" #20 daemon prio=5 os_prio=0 tid=0x18390400 nid=0x7d0 waiting on condition [0x18d4f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a044918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-3" #12 prio=10 os_prio=2 tid=0x154b4c00 nid=0x1ba0 runnable [0x1730f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02e818> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a03b060> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02e7a0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-2" #11 prio=10 os_prio=2 tid=0x154a4800 nid=0x1524 runnable [0x1735f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02e400> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a038df8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02e388> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-2" #19 daemon prio=5 os_prio=0 tid=0x15374000 nid=0x1748 waiting on condition [0x14f9f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a0447a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-3-1" #18 daemon prio=1 os_prio=-2 tid=0x15336c00 nid=0x1ffc waiting on condition [0x15dcf000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.couchbase.client.deps.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-1" #10 prio=10 os_prio=2 tid=0x152f4400 nid=0x1934 runnable [0x15c3f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02dfe8> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a036b90> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02df70> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-1" #16 daemon prio=5 os_prio=0 tid=0x15284800 nid=0x180c waiting on condition [0x15adf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a044860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00f74c00 nid=0xefc runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x00f65800 nid=0x1600 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x00f5f800 nid=0x1a78 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x00f5cc00 nid=0x1e54 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00f20c00 nid=0x1e30 in Object.wait() [0x14cff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x09de3798> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
        - locked <0x09de3798> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00f1a800 nid=0x1154 in Object.wait() [0x047bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x09de3938> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x09de3938> (a java.lang.ref.Reference$Lock)

"VM Thread" os_prio=2 tid=0x00f17400 nid=0x1d7c runnable

"VM Periodic Task Thread" os_prio=2 tid=0x00f7e400 nid=0x1264 waiting on condition

JNI global references: 242


Code to reproduce:

    final ViewQuery view = ViewQuery
        .from("views", "myView")
        .startKey(JsonArray.from(5))
        .endKey(JsonArray.from(6))
        .reduce(false);

    logger.debug("view=" + view);

    final Observable<ViewResult> query = bucket.query(view);
    logger.debug("query=" + query);

    query
        .doOnNext(item -> logger.debug("item={}; item.success()={}", item, item.success()))
        .filter(item -> item.success())
        .doOnNext(item -> logger.debug("getting rows"))
        .flatMap(ViewResult::rows)
        .doOnNext(item -> logger.debug("view row={}", item))
        .filter(item -> choose(item))
        .take(howMany)
        .doOnNext(item -> log(item))
        .doOnError(e -> e.printStackTrace())
        .doOnTerminate(() -> {
          System.out.println("done");
          cluster.disconnect();
        })
        .subscribe();
    logger.debug("subscribed");

    Thread.sleep(1000);
    cluster.disconnect();

(This is called by the main thread after connecting to the cluster)

This will print done, but never terminate.

 Comments   
Comment by Michael Nitschinger [ 18/Aug/14 ]
thanks for reporting it, I'll look into it soon.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Found the issue, from the client the stateful environment was not properly shutdown. Should be fixed on master and is in the beta release. Let me know if it still not works, thanks!
Comment by Adam Honen [ 24/Aug/14 ]
Actually it did not work.
I tried with the beta jar, but I still see those Netty IO (non daemon) threads hanging around.

I tried also adding a call to bucket.close(), but that didn't help either.

Comment by Adam Honen [ 24/Aug/14 ]
Attached is the TRACE log that is emitted after I call the following two lines of code:

bucket.close();
cluster.disconnect();
Comment by Michael Nitschinger [ 27/Aug/14 ]
Ah, are you passing in a custom environment? Your demo code doesn't show the init path. If so, you need to shutdown() it manually. We can't do that on cluster.disconnect() because it could be shared across more cluster objects.
Comment by Michael Nitschinger [ 27/Aug/14 ]
Note that I made a change for beta2 so that 1) the owned pools are properly named (prefixed with cb) and are all daemonized, so even if the user doesn't call shutdown as he should at least the JVM exits.




[JCBC-508] PersistTo.TWO doesn't fail with single node cluster. Created: 11/Aug/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In the observe poll part, the persist counter gets decremented which leads to a non-failing op on a single node cluster with PersistTo.TWO




[JCBC-507] Provide compatibility with 3.0 spatial views Created: 11/Aug/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In 3.0, the returned result doesn't contain a "bbox" property anymore. It was renamed to "key" to be more along the lines of the mapreduce views. The structure also changes a bit. The "bbox" was an array with 4 elements, with [min_x, min_y, max_x, max_y]. The key is an array containing the bounds of the dimensions, so it is: [[min_x, max_x], [min_x, max_y]].




[JCBC-505] Concurrency issue in the View Query builder with regexes Created: 30/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-504] Couchbase hangs during tomcat shutdown Created: 30/Jul/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2, 1.4.3
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Major
Reporter: Benoit Beaudet Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Tomcat 7.0.45.


 Description   
Using couchbase lib inside a WAR. All clients connections are stopped by the application but Tomcat can't stop.
Tomcat stop process is blocked by a CouchbaseConfigConnection thread (deamon=false)

{code}
  java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    - locked <0x0000000795b80af0> (a sun.nio.ch.Util$2)
    - locked <0x0000000795b80b00> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000795b80aa8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:420)
    at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:321)
{code}


Looking a the bootstrap code from com.couchbase.client.vbucket.provider.BucketConfigurationProvider class.
A CouchbaseConfigConnection is created but not closed if everything is working fine.
{code}
private boolean tryBinaryBootstrapForNode(InetSocketAddress node)
    throws Exception

  connection = new CouchbaseConfigConnection(
        cf.getReadBufSize(), fact, Collections.singletonList(node),
        initialObservers, cf.getFailureMode(),
        cf.getOperationFactory()
      );

{code}



Hack to resolve the issue.
{code}
  private void killConfigConnection(){
        ThreadGroup rootGroup = Thread.currentThread( ).getThreadGroup( );
        ThreadGroup parentGroup;
        while ( ( parentGroup = rootGroup.getParent() ) != null ) {
            rootGroup = parentGroup;
        }

        Thread[] threads = new Thread[ rootGroup.activeCount() ];
        while ( rootGroup.enumerate( threads, true ) == threads.length ) {
            threads = new Thread[ threads.length * 2 ];
        }

        for (int i = 0; i < threads.length; i++){
            final Thread thread = threads[i];
            if(thread instanceof CouchbaseConfigConnection){
                final CouchbaseConfigConnection configConn = (CouchbaseConfigConnection) thread;
                try {
                    configConn.shutdown();
                } catch (final IOException e) {
                   //
                }
            }

        }
   
{code}

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
Is that a duplicate of 503?
Comment by Benoit Beaudet [ 04/Aug/14 ]
This issue isn't a duplicate. A non deamon thread is still active in Tomcat. The CouchbaseConfigConnection used at boostrap should be shutdown.
Comment by Michael Nitschinger [ 05/Aug/14 ]
Hmm maybe I don't understand it. Do you call client.shutdown() or just shutdown tomcat? Because they are by intention non-daemon threads and you need to shutdown the client manually. or do you shut it down but some resources are not freed?
Comment by Benoit Beaudet [ 05/Aug/14 ]
I call client.shutdown in web context destroy event, but the CouchbaseConfigConnection isn't under my control and not freed.




[JCBC-503] Client shutdown doesn't release observers and leak memory. Created: 29/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: Benoit Beaudet Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If a client is using VBucket the shutdown method call BucketConfigurationProvider.shutdown().

{code}
try {
      CouchbaseConnectionFactory cf = (CouchbaseConnectionFactory) connFactory;
      cf.getConfigurationProvider().shutdown();
      if(vconn != null) {
        vconn.shutdown();
      }
{code}

The observers array reference closed CoucheBaseClient.
To fix the issue, I need to call directly unsubscribe method.
connectionFactory.getConfigurationProvider().unsubscribe((Reconfigurable) client);
 

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
Does that look valid?
http://review.couchbase.org/#/c/40263/
Clearing them on shutdown seems valuable
Comment by Benoit Beaudet [ 04/Aug/14 ]
Hi!

I think that the observers list is global. A shutdown of 1 client will clears all active clients observers?.
This is why I used unsubscribe.

 @Override
  public void unsubscribe(Reconfigurable rec) {
    observers.remove(rec);
  }
Comment by Michael Nitschinger [ 05/Aug/14 ]
Hey,

the thing is if you shut down the config manager noone else can use it anyway because its down, so we can just unsubscribe all of them right in the shutdown method. There is no need to keep more of them around, especially because this is intended to be internal interface.
Comment by Benoit Beaudet [ 05/Aug/14 ]
In my scenario. I use a pool to create more connections under load and release them after an idle period.
I shutdown clients not only at web app shutdown, but also at runtime.

The factory is a singleton in my project. For each CouchbaseClientConnection created with the factory, an observer is added.
cf.getConfigurationProvider().subscribe(this);

CouchbaseClient.shutdown method call indirectly the factory getConfigurationProvider().
   CouchbaseConnectionFactory cf = (CouchbaseConnectionFactory) connFactory;
      cf.getConfigurationProvider().shutdown();

This is why is suggested that observers array size should match created connections. Using internal subscribe/unsubscribe to match the pattern.



Comment by Michael Nitschinger [ 05/Aug/14 ]
Every time you create a new CouchbaseClient object, all of those objects are created again.

You should never ever shut down or listen on the configuration provider on your own, that is internal behavior. You should 1) always only use the CouchbaseClient interface and 2) reuse them as long as possible.

Why do you shut them down at runtime?
Comment by Benoit Beaudet [ 05/Aug/14 ]
I use CouchbaseClient.shutdown() to evict clients and use new CouchbaseClient(using the singleton CouchbaseConnectionFactory as parameter) to recreate them. I shouldn't use the Configuration provider directly.
But the internal configuration provider is keeping references on all connections created, even if the pool closed them. I create more than 1 client at bootup and close it at shutdown.
The pool manage the growth and idling clients. It's the reason to shutdown a client if idling for a long time.




[JCBC-502] Sometimes couchbase client stuck Created: 29/Jul/14  Updated: 21/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 2.0-dp2
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Mac OSX as client


 Description   
Sometimes I run the client like in the examples in the community blog and the client stuck and Doesn't return. To validate that I run the client several times one after the the other and sometimes it get stuck and some don't

 Comments   
Comment by Michael Nitschinger [ 11/Aug/14 ]
Can you provide logging with level set to FINEST? that would be of great help, thanks.
Comment by Oded Hassidi [ 12/Aug/14 ]
I am running my test in a main, and I have debug level entries. Is that enough?
Comment by Michael Nitschinger [ 12/Aug/14 ]
Yes send over whatever you have, and the code would also be good.
Comment by Oded Hassidi [ 12/Aug/14 ]
Code:


public class Main {
public static void main(String[] args) throws IOException {
      
        System.setProperty("com.couchbase.client.queryEnabled", "true");

        CouchbaseCluster cc = new CouchbaseCluster("obbe-centos3.tlv.lpnet.com", "obbe-centos1.tlv.lpnet.com", "obbe-centos2.tlv.lpnet.com");

        final Bucket[] bucket = new CouchbaseBucket[1];
        System.out.println("******** Opening the Bucket");
        cc.openBucket("visitors-odedh").subscribe(b -> {
            System.out.println("################### Bucket is opened: ");
                System.out.println(b.toString());
                bucket[0] = b;
        });


        System.out.println("enter something: ");
        System.in.read();
    }
}



log:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -Didea.launcher.port=7534 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 14 EAP.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dal/target/classes:/Users/odedh/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar:/Users/odedh/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar:/Users/odedh/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dao/target/classes:/Users/odedh/.m2/repository/com/liveperson/framework/rest-api-data-objects/2.0.2.11/rest-api-data-objects-2.0.2.11.jar:/Users/odedh/.m2/repository/com/liveperson/RestApiCommon/1.0.2-6/RestApiCommon-1.0.2-6.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-core/1.9.1/jersey-core-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-client/1.9.1/jersey-client-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-server/1.9.1/jersey-server-1.9.1.jar:/Users/odedh/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-json/1.9.1/jersey-json-1.9.1.jar:/Users/odedh/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/odedh/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/odedh/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/odedh/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/odedh/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/odedh/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.3/jackson-core-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.3/jackson-mapper-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/Users/odedh/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar:/Users/odedh/.m2/repository/org/springframework/spring-context/4.0.6.RELEASE/spring-context-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-aop/4.0.6.RELEASE/spring-aop-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/odedh/.m2/repository/org/springframework/spring-beans/4.0.6.RELEASE/spring-beans-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-core/4.0.6.RELEASE/spring-core-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-expression/4.0.6.RELEASE/spring-expression-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/couchbase/client/couchbase-client/2.0.0-dp2/couchbase-client-2.0.0-dp2.jar:/Users/odedh/.m2/repository/com/couchbase/client/core-io/0.2/core-io-0.2.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.4.0/jackson-databind-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.0/jackson-annotations-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.0/jackson-core-2.4.0.jar:/Users/odedh/.m2/repository/com/lmax/disruptor/3.2.1/disruptor-3.2.1.jar:/Users/odedh/.m2/repository/com/netflix/rxjava/rxjava-core/0.19.2/rxjava-core-0.19.2.jar:/Users/odedh/.m2/repository/com/typesafe/config/1.2.1/config-1.2.1.jar:/Users/odedh/.m2/repository/io/netty/netty-all/4.0.21.Final/netty-all-4.0.21.Final.jar:/Users/odedh/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/odedh/.m2/repository/org/springframework/spring-test/4.0.6.RELEASE/spring-test-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring/4.0.0.1/SnmpMonitoring-4.0.0.1.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring-api/4.0.0.1/SnmpMonitoring-api-4.0.0.1.jar:/Users/odedh/.m2/repository/org/apache/xmlbeans/xmlbeans/2.4.0/xmlbeans-2.4.0.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus/3.0.0.2/ServiceStatus-3.0.0.2.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus-API/3.0.0.2/ServiceStatus-API-3.0.0.2.jar:/Applications/IntelliJ IDEA 14 EAP.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.liveperson.visitorfeed.dal.couchbase.CouchbaseVisitorDao
15:55:05.598 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
15:55:05.604 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 733553586
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/5f/wk7ckrbx1f30521n8rtxndzwnvk8xk/T (java.io.tmpdir)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
15:55:05.712 [main] DEBUG c.c.c.c.config.ConfigurationProvider - Setting seed hosts to [obbe-centos1.tlv.lpnet.com/192.168.15.120, obbe-centos3.tlv.lpnet.com/192.168.15.190, obbe-centos2.tlv.lpnet.com/192.168.15.2]
******** Opening the Bucket
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
enter something:
15:55:05.780 [RxComputationThreadPool-1] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x9729d41313197dd7
15:55:05.785 [RxComputationThreadPool-1] DEBUG i.n.channel.ChannelOutboundBuffer - -Dio.netty.threadLocalDirectBufferSize: 65536
15:55:05.786 [RxComputationThreadPool-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
15:55:05.795 [RxComputationThreadPool-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
15:55:05.880 [nioEventLoopGroup-2-2] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple
15:55:05.981 [nioEventLoopGroup-2-2] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-6] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-3] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.984 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.984 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.984 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos1.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos3.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.985 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos2.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-1] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.994 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:05.996 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:06.238 [RxComputationThreadPool-2] DEBUG c.c.c.c.config.ConfigurationProvider - Applying new configuration DefaultCouchbaseBucketConfig{name='visitors-odedh', locator=null, uri='/pools/default/buckets/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', streamingUri='/pools/default/bucketsStreaming/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', nodeInfo=[NodeInfo{viewUri='http://obbe-centos1.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos1.tlv.lpnet.com/192.168.15.120, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos2.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos2.tlv.lpnet.com/192.168.15.2, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos3.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos3.tlv.lpnet.com/192.168.15.190, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}], partitionInfo=PartitionInfo{numberOfReplicas=1, partitionHosts=[obbe-centos1.tlv.lpnet.com, obbe-centos2.tlv.lpnet.com, obbe-centos3.tlv.lpnet.com], partitions=[[m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]]], forwardPartitions=[]}, tainted=false, rev=2070}
15:55:06.256 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8091
15:55:06.257 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos2.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8091
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos1.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8092
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos2.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8092
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos3.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8092
15:55:06.259 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8093
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8093
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos2.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8093
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8091
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
Comment by Michael Nitschinger [ 21/Aug/14 ]
Hi,

can you please retry with dp3 and see if the issue still persists? If so, please also provide the debug logs - thanks! Also, can you specify what "hangs" means? Your println does never show?
Comment by Oded Hassidi [ 24/Aug/14 ]
Thanks, yes my println doesn't ever show!
Will check it with the beta, sorry for the delay I was OOO couple of days :)
Comment by Michael Nitschinger [ 04/Sep/14 ]
Hey, any update on this?
Comment by Michael Nitschinger [ 15/Sep/14 ]
I've removed the fix version since it's not clear if its a bug or not anymore. Please provide any details if you can as soon as possible so we can fix it if needed thanks!
Comment by Oded Hassidi [ 21/Sep/14 ]
Hi Michael and thanks
Sorry for not being responsive in the last few weeks

I will try to check this and respond with latest findings in the next couple of days

Thanks again




[JCBC-501] Implement Support for DCP Created: 29/Jul/14  Updated: 29/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Implement support for 3.0+ DCP as a replacement for the TAP protocol and TAP Client.






[JCBC-500] CouchbaseClient doesn't timeout, hangs forever Created: 27/Jul/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Milan Simonovic Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
On a single node cluster:

1. open CouchbaseClient with opTimeout = 10sec (on a different machine)
2. write 50 docs (works fine)
3. make the server drop packets for 15sec (iptables -A INPUT -p tcp --dport 200:65535 -j DROP)
4. try to write another doc -> blocks forever

I've tried different timeout values: opTimeout =2.5sec, dropping packets for 5sec, but it also never times-out.

The code to reproduce this issue is here: https://github.com/mbsimonovic/jepsen-couchbase.
A more detailed explanation in my blog post: http://milansimonovic.com/2014/07/22/smashing-couchbase/

 Comments   
Comment by Milan Simonovic [ 28/Jul/14 ]
tested with Java SDK v1.4.3
Comment by Michael Nitschinger [ 14/Aug/14 ]
Hey Milan,

I cloned your project and adjusted it to my 3 node cluster. What I got was:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -Didea.launcher.port=7540 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 13.app/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/michael/tmp/jepsen-couchbase/target/classes:/Users/michael/.m2/repository/com/couchbase/client/couchbase-client/1.4.3/couchbase-client-1.4.3.jar:/Users/michael/.m2/repository/io/netty/netty/3.5.5.Final/netty-3.5.5.Final.jar:/Users/michael/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/michael/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/michael/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/michael/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore-nio/4.3/httpcore-nio-4.3.jar:/Users/michael/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/michael/.m2/repository/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/michael/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/michael/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/michael/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Applications/IntelliJ IDEA 13.app/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.milansimonovic.jepsen.CouchTimeoutTest
2014-08-14 10:19:38,106 INFO [com.milansimonovic.jepsen.RemoteConnection] - 192.168.56.101
2014-08-14 10:19:38,636 INFO [com.milansimonovic.jepsen.CouchbaseClientFactory] - CouchbaseClientFactory{192.168.56.101, default, }
2014-08-14 10:19:38,783 INFO [com.couchbase.client.vbucket.provider.BucketConfigurationProvider] - Could bootstrap through carrier publication.
2014-08-14 10:19:38,800 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,800 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.102:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,801 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.103:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,806 INFO [com.couchbase.client.CouchbaseClient] - CouchbaseConnectionFactory{bucket='default', nodes=[http://192.168.56.101:8091/pools], order=RANDOM, opTimeout=10000, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=1000, obsTimeout=10000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-08-14 10:19:38,849 INFO [com.couchbase.client.CouchbaseClient] - viewmode property isn't defined. Setting viewmode to production mode
2014-08-14 10:19:38,849 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 10:19:39,869 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:39,870 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 10:19:39,919 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 1 in 43
2014-08-14 10:19:39,922 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 2 in 3
2014-08-14 10:19:39,923 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 3 in 1
2014-08-14 10:19:39,925 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 4 in 2
2014-08-14 10:19:39,927 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 5 in 2
2014-08-14 10:19:39,928 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 6 in 1
2014-08-14 10:19:39,930 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 7 in 2
2014-08-14 10:19:39,931 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 8 in 1
2014-08-14 10:19:39,933 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 9 in 2
2014-08-14 10:19:39,934 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 10 in 1
2014-08-14 10:19:39,936 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 11 in 2
2014-08-14 10:19:39,938 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 12 in 2
2014-08-14 10:19:39,939 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 13 in 1
2014-08-14 10:19:39,941 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 14 in 2
2014-08-14 10:19:39,943 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 15 in 2
2014-08-14 10:19:39,945 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 16 in 2
2014-08-14 10:19:39,946 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 17 in 1
2014-08-14 10:19:39,948 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 18 in 2
2014-08-14 10:19:39,949 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 19 in 1
2014-08-14 10:19:39,953 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 20 in 2
2014-08-14 10:19:39,955 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 21 in 2
2014-08-14 10:19:39,956 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 22 in 1
2014-08-14 10:19:39,958 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 23 in 2
2014-08-14 10:19:39,959 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 24 in 1
2014-08-14 10:19:39,961 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 25 in 2
2014-08-14 10:19:39,963 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 26 in 2
2014-08-14 10:19:39,964 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 27 in 1
2014-08-14 10:19:39,966 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 28 in 2
2014-08-14 10:19:39,968 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 29 in 2
2014-08-14 10:19:39,969 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 30 in 1
2014-08-14 10:19:39,971 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 31 in 2
2014-08-14 10:19:39,972 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 32 in 1
2014-08-14 10:19:39,974 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 33 in 2
2014-08-14 10:19:39,976 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 34 in 2
2014-08-14 10:19:39,977 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 35 in 1
2014-08-14 10:19:39,979 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 36 in 2
2014-08-14 10:19:39,981 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 37 in 2
2014-08-14 10:19:39,982 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 38 in 1
2014-08-14 10:19:39,984 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 39 in 2
2014-08-14 10:19:39,986 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 40 in 2
2014-08-14 10:19:39,988 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 41 in 1
2014-08-14 10:19:39,989 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 42 in 1
2014-08-14 10:19:39,991 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 43 in 1
2014-08-14 10:19:39,992 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 44 in 1
2014-08-14 10:19:39,993 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 45 in 1
2014-08-14 10:19:39,995 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 46 in 2
2014-08-14 10:19:39,996 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 47 in 1
2014-08-14 10:19:39,998 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 48 in 2
2014-08-14 10:19:39,999 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 49 in 1
2014-08-14 10:19:40,001 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 50 in 2
2014-08-14 10:19:40,001 INFO [com.milansimonovic.jepsen.Cluster] - creating 1 way partition
2014-08-14 10:19:41,003 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:41,003 INFO [com.milansimonovic.jepsen.Cluster] - creating partition DONE
2014-08-14 10:19:41,004 INFO [com.milansimonovic.jepsen.Cluster] - scheduled heal in 15 seconds
2014-08-14 10:19:56,005 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 10:19:57,007 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:57,008 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 10:19:57,008 INFO [com.milansimonovic.jepsen.Cluster] - closing connections

and the doc on the server looked like:

{
  "elements": [
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10,
    11,
    12,
    13,
    14,
    15,
    16,
    17,
    18,
    19,
    20,
    21,
    22,
    23,
    24,
    25,
    26,
    27,
    28,
    29,
    30,
    31,
    32,
    33,
    34,
    35,
    36,
    37,
    38,
    39,
    40,
    41,
    42,
    43,
    44,
    45,
    46,
    47,
    48,
    49,
    50
  ]
}

I wonder what I did different? Does it happen for you only in the single-node case? Or does running the test just not cover the bug?
Comment by Michael Nitschinger [ 14/Aug/14 ]
Ah now I get it, there should be much more appended than just 50.
Comment by Michael Nitschinger [ 14/Aug/14 ]
Ah I fixed your code. Let me do a PR for your github project.

You need to move the gets into the try/catch block as well since it can also throw a timeout (and will if there is a partition):

                CASValue<Object> casValue = client.gets(key);
                String updatedValue = appendElement(casValue.getValue().toString(), nextVal);

When I move it inside I get:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:61775,suspend=y,server=n -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/michael/tmp/jepsen-couchbase/target/classes:/Users/michael/.m2/repository/com/couchbase/client/couchbase-client/1.4.3/couchbase-client-1.4.3.jar:/Users/michael/.m2/repository/io/netty/netty/3.5.5.Final/netty-3.5.5.Final.jar:/Users/michael/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/michael/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/michael/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/michael/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore-nio/4.3/httpcore-nio-4.3.jar:/Users/michael/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/michael/.m2/repository/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/michael/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/michael/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/michael/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Applications/IntelliJ IDEA 13.app/lib/idea_rt.jar" com.milansimonovic.jepsen.CouchTimeoutTest
Connected to the target VM, address: '127.0.0.1:61775', transport: 'socket'
2014-08-14 11:02:47,408 INFO [com.milansimonovic.jepsen.RemoteConnection] - 192.168.56.101
2014-08-14 11:02:47,771 INFO [com.milansimonovic.jepsen.CouchbaseClientFactory] - CouchbaseClientFactory{192.168.56.101, default, }
2014-08-14 11:02:47,908 INFO [com.couchbase.client.vbucket.provider.BucketConfigurationProvider] - Could bootstrap through carrier publication.
2014-08-14 11:02:47,920 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 11:02:47,923 INFO [com.couchbase.client.CouchbaseClient] - CouchbaseConnectionFactory{bucket='default', nodes=[http://192.168.56.101:8091/pools], order=RANDOM, opTimeout=10000, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=1000, obsTimeout=10000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-08-14 11:02:47,956 INFO [com.couchbase.client.CouchbaseClient] - viewmode property isn't defined. Setting viewmode to production mode
2014-08-14 11:02:47,956 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:02:48,970 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:48,970 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
counting down 943596453
counting down 99276090
2014-08-14 11:02:49,042 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 1 in 61
counting down 1076537872
2014-08-14 11:02:49,048 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 2 in 6
2014-08-14 11:02:49,054 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 3 in 6
counting down 1413217636
2014-08-14 11:02:49,060 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 4 in 6
counting down 491904845
2014-08-14 11:02:49,066 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 5 in 6
counting down 1312922328
2014-08-14 11:02:49,072 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 6 in 6
counting down 1529791274
2014-08-14 11:02:49,078 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 7 in 6
counting down 1349895670
2014-08-14 11:02:49,083 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 8 in 5
counting down 1129169666
2014-08-14 11:02:49,089 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 9 in 6
counting down 26740457
counting down 746758755
2014-08-14 11:02:49,094 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 10 in 5
counting down 1386645616
2014-08-14 11:02:49,100 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 11 in 5
2014-08-14 11:02:49,106 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 12 in 6
counting down 2145256693
2014-08-14 11:02:49,111 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 13 in 5
counting down 1051781801
2014-08-14 11:02:49,117 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 14 in 6
counting down 119068326
2014-08-14 11:02:49,123 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 15 in 6
counting down 1637559101
2014-08-14 11:02:49,128 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 16 in 5
counting down 1861050967
2014-08-14 11:02:49,134 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 17 in 6
counting down 1462839425
2014-08-14 11:02:49,140 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 18 in 6
counting down 383609127
2014-08-14 11:02:49,147 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 19 in 6
counting down 774334978
2014-08-14 11:02:49,156 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 20 in 8
counting down 735922763
2014-08-14 11:02:49,162 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 21 in 6
counting down 1970674321
2014-08-14 11:02:49,170 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 22 in 7
counting down 136298825
2014-08-14 11:02:49,177 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 23 in 7
counting down 494783794
2014-08-14 11:02:49,185 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 24 in 8
counting down 1546867652
2014-08-14 11:02:49,192 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 25 in 7
counting down 433354817
2014-08-14 11:02:49,199 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 26 in 7
counting down 1865938875
2014-08-14 11:02:49,207 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 27 in 8
counting down 483439410
2014-08-14 11:02:49,214 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 28 in 7
counting down 1513848351
2014-08-14 11:02:49,221 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 29 in 7
counting down 1612120937
2014-08-14 11:02:49,228 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 30 in 7
counting down 810400511
2014-08-14 11:02:49,235 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 31 in 7
counting down 689821415
2014-08-14 11:02:49,242 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 32 in 7
counting down 685219421
2014-08-14 11:02:49,248 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 33 in 6
counting down 1887689829
counting down 759433928
2014-08-14 11:02:49,254 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 34 in 6
2014-08-14 11:02:49,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 35 in 5
counting down 1781091177
2014-08-14 11:02:49,266 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 36 in 7
counting down 2123891975
2014-08-14 11:02:49,271 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 37 in 5
counting down 278306373
2014-08-14 11:02:49,278 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 38 in 6
counting down 495568085
2014-08-14 11:02:49,284 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 39 in 6
counting down 1029420558
2014-08-14 11:02:49,290 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 40 in 5
counting down 55913631
2014-08-14 11:02:49,296 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 41 in 6
counting down 664141318
2014-08-14 11:02:49,302 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 42 in 6
counting down 1953048909
2014-08-14 11:02:49,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 43 in 5
counting down 350362120
counting down 1048178143
2014-08-14 11:02:49,313 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 44 in 6
2014-08-14 11:02:49,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 45 in 5
counting down 907297240
2014-08-14 11:02:49,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 46 in 5
counting down 193602838
2014-08-14 11:02:49,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 47 in 6
counting down 255614005
2014-08-14 11:02:49,335 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 48 in 6
counting down 2128240408
2014-08-14 11:02:49,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 49 in 5
counting down 1783946138
2014-08-14 11:02:49,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 50 in 6
2014-08-14 11:02:49,347 INFO [com.milansimonovic.jepsen.Cluster] - creating 1 way partition
2014-08-14 11:02:50,348 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:50,348 INFO [com.milansimonovic.jepsen.Cluster] - creating partition DONE
2014-08-14 11:02:50,349 INFO [com.milansimonovic.jepsen.Cluster] - scheduled heal in 15 seconds
counting down 1330247343
2014-08-14 11:03:03,213 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - timed out adding 51
2014-08-14 11:03:03,214 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - error adding 51
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1147)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1211)
at com.milansimonovic.jepsen.CouchTimeoutTest.addElementToArray(CouchTimeoutTest.java:65)
at com.milansimonovic.jepsen.CouchTimeoutTest.appendToDocument(CouchTimeoutTest.java:102)
at com.milansimonovic.jepsen.CouchTimeoutTest.run(CouchTimeoutTest.java:49)
at com.milansimonovic.jepsen.CouchTimeoutTest.main(CouchTimeoutTest.java:39)
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 192.168.56.101/192.168.56.101:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1137)
... 5 more
2014-08-14 11:03:05,349 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:03:06,351 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - closing connections
counting down 1330247343
counting down 808247635
counting down 1562173079
2014-08-14 11:03:08,230 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 52 in 5014
counting down 1897031859
2014-08-14 11:03:08,236 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 53 in 6
2014-08-14 11:03:08,241 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 54 in 5
counting down 1029104896
2014-08-14 11:03:08,247 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 55 in 6
counting down 1789373824
2014-08-14 11:03:08,252 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 56 in 5
counting down 1374957093
2014-08-14 11:03:08,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 57 in 7
counting down 859935576
2014-08-14 11:03:08,265 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 58 in 5
counting down 959778517
2014-08-14 11:03:08,270 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 59 in 5
counting down 1856597982
2014-08-14 11:03:08,276 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 60 in 6
counting down 509403432
2014-08-14 11:03:08,283 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 61 in 7
counting down 565734120
2014-08-14 11:03:08,289 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 62 in 6
counting down 812288940
2014-08-14 11:03:08,295 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 63 in 6
counting down 362337760
2014-08-14 11:03:08,301 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 64 in 6
counting down 1815360474
counting down 238680990
2014-08-14 11:03:08,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 65 in 6
2014-08-14 11:03:08,312 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 66 in 5
counting down 1793734174
2014-08-14 11:03:08,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 67 in 6
counting down 965663094
2014-08-14 11:03:08,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 68 in 5
counting down 1961465726
2014-08-14 11:03:08,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 69 in 6
counting down 894265206
counting down 237825287
2014-08-14 11:03:08,334 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 70 in 5
2014-08-14 11:03:08,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 71 in 6
counting down 1093062877
2014-08-14 11:03:08,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 72 in 6
counting down 2043477472
2014-08-14 11:03:08,352 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 73 in 6
counting down 660886392
2014-08-14 11:03:08,358 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 74 in 6
counting down 1321850438
2014-08-14 11:03:08,363 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 75 in 5
counting down 966674298
2014-08-14 11:03:08,368 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 76 in 5
counting down 665030228
counting down 584709707
2014-08-14 11:03:08,373 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 77 in 5
2014-08-14 11:03:08,379 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 78 in 6
counting down 581070768
2014-08-14 11:03:08,384 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 79 in 5
counting down 1766611209
2014-08-14 11:03:08,390 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 80 in 6
counting down 1378265946
2014-08-14 11:03:08,395 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 81 in 5
counting down 1067480315
2014-08-14 11:03:08,400 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 82 in 5
counting down 240010064
2014-08-14 11:03:08,406 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 83 in 6
counting down 96589697
counting down 1650118680
2014-08-14 11:03:08,411 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 84 in 5
2014-08-14 11:03:08,417 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 85 in 6
counting down 1583441668
2014-08-14 11:03:08,422 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 86 in 5
counting down 575269723
2014-08-14 11:03:08,427 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 87 in 5
counting down 692018508
2014-08-14 11:03:08,434 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 88 in 6
counting down 500269667
2014-08-14 11:03:08,440 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 89 in 6
counting down 1214091162
2014-08-14 11:03:08,446 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 90 in 6
counting down 631482105
2014-08-14 11:03:08,452 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 91 in 6
counting down 1982159420
2014-08-14 11:03:08,458 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 92 in 6
counting down 1744182273
counting down 1131642426
2014-08-14 11:03:08,463 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 93 in 5
2014-08-14 11:03:08,469 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 94 in 6
counting down 2069859523
2014-08-14 11:03:08,475 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 95 in 6
counting down 662166830
2014-08-14 11:03:08,481 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 96 in 6
counting down 1882643413
2014-08-14 11:03:08,487 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 97 in 6
counting down 678230343
2014-08-14 11:03:08,493 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 98 in 6
counting down 793919773
2014-08-14 11:03:08,498 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 99 in 5
counting down 911566708
2014-08-14 11:03:08,505 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 100 in 7
counting down 1887173144
2014-08-14 11:03:08,510 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 101 in 5
counting down 1058177937
2014-08-14 11:03:08,516 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 102 in 6
counting down 41252807
2014-08-14 11:03:08,523 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 103 in 6
counting down 528932410
2014-08-14 11:03:08,531 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 104 in 8
counting down 805461224
2014-08-14 11:03:08,538 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 105 in 7
counting down 463949682
2014-08-14 11:03:08,546 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 106 in 8
counting down 851822734
2014-08-14 11:03:08,554 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 107 in 8
counting down 454797282
2014-08-14 11:03:08,561 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 108 in 7
counting down 962354549
2014-08-14 11:03:08,568 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 109 in 7
counting down 413223411
2014-08-14 11:03:08,574 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 110 in 6
counting down 79455185
2014-08-14 11:03:08,580 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 111 in 6
counting down 1113518099
2014-08-14 11:03:08,586 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 112 in 6
counting down 431139544
2014-08-14 11:03:08,592 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 113 in 6
counting down 198084300
2014-08-14 11:03:08,599 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 114 in 7
counting down 1821101711
2014-08-14 11:03:08,605 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 115 in 6
counting down 1499489637
2014-08-14 11:03:08,611 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 116 in 6
counting down 1435037947
2014-08-14 11:03:08,618 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 117 in 7
counting down 705688672
2014-08-14 11:03:08,624 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 118 in 6
counting down 407021592
2014-08-14 11:03:08,630 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 119 in 6
counting down 1121168696
2014-08-14 11:03:08,637 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 120 in 7
counting down 1347326641
counting down 87539756
2014-08-14 11:03:08,642 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 121 in 5
2014-08-14 11:03:08,647 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 122 in 5
counting down 137935610
2014-08-14 11:03:08,653 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 123 in 6
counting down 1474133160
2014-08-14 11:03:08,659 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 124 in 6
counting down 1931651064
2014-08-14 11:03:08,665 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 125 in 6
counting down 1211051792
2014-08-14 11:03:08,688 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 126 in 23
counting down 93172613
2014-08-14 11:03:08,693 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 127 in 5
counting down 965322656
2014-08-14 11:03:08,699 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 128 in 6
counting down 1601358350
2014-08-14 11:03:08,705 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 129 in 6
counting down 1855207775
2014-08-14 11:03:08,710 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 130 in 5
counting down 107851411
2014-08-14 11:03:08,716 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 131 in 6
counting down 1915600217
2014-08-14 11:03:08,721 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 132 in 5
counting down 1002847144
2014-08-14 11:03:08,727 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 133 in 6
counting down 1960289102
2014-08-14 11:03:08,732 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 134 in 5
counting down 530940529
2014-08-14 11:03:08,738 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 135 in 6
counting down 16593833
2014-08-14 11:03:08,744 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 136 in 6
counting down 183255909
2014-08-14 11:03:08,749 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 137 in 5
counting down 1139297445
counting down 1886255267
2014-08-14 11:03:08,755 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 138 in 5
2014-08-14 11:03:08,760 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 139 in 5
counting down 426044796
2014-08-14 11:03:08,765 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 140 in 5
counting down 1903915614
2014-08-14 11:03:08,770 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 141 in 5
counting down 1895269808
2014-08-14 11:03:08,776 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 142 in 6
counting down 1866519533
counting down 1366412404
2014-08-14 11:03:08,781 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 143 in 5
2014-08-14 11:03:08,786 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 144 in 5
counting down 2075311494
2014-08-14 11:03:08,792 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 145 in 6
counting down 925138245
2014-08-14 11:03:08,797 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 146 in 5
counting down 1578242039
2014-08-14 11:03:08,802 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 147 in 5
counting down 23034413
2014-08-14 11:03:08,808 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 148 in 6
counting down 1869588749
counting down 2067087344
2014-08-14 11:03:08,814 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 149 in 6
2014-08-14 11:03:08,820 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 150 in 6
counting down 899087062
2014-08-14 11:03:08,826 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 151 in 6
counting down 1340235867
2014-08-14 11:03:08,832 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 152 in 6
counting down 1574412139
2014-08-14 11:03:08,838 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 153 in 6
counting down 256897438
2014-08-14 11:03:08,844 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 154 in 6
counting down 227404523
2014-08-14 11:03:08,849 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 155 in 5
counting down 1370600377
2014-08-14 11:03:08,855 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 156 in 5
counting down 821838371
2014-08-14 11:03:08,861 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 157 in 6
counting down 1087346339
counting down 578287331
2014-08-14 11:03:08,866 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 158 in 5
2014-08-14 11:03:08,872 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 159 in 6
counting down 2114430319
2014-08-14 11:03:08,877 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 160 in 5
counting down 750263766
2014-08-14 11:03:08,883 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 161 in 6
counting down 285142124
2014-08-14 11:03:08,889 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 162 in 6
counting down 228835454
2014-08-14 11:03:08,895 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 163 in 6
counting down 1322000504
2014-08-14 11:03:08,900 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 164 in 5
counting down 578208474
2014-08-14 11:03:08,906 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 165 in 6
counting down 1158469492
counting down 1446993819
2014-08-14 11:03:08,911 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 166 in 5
2014-08-14 11:03:08,917 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 167 in 6
counting down 2036300484
2014-08-14 11:03:08,923 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 168 in 5
counting down 1839929127
2014-08-14 11:03:08,928 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 169 in 5
counting down 1321196177
2014-08-14 11:03:08,934 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 170 in 6
counting down 1657833306
2014-08-14 11:03:08,941 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 171 in 7
counting down 1616599198
2014-08-14 11:03:08,948 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 172 in 7
counting down 1434216861
2014-08-14 11:03:08,955 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 173 in 7
counting down 1697078001
2014-08-14 11:03:08,962 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 174 in 7
counting down 53209944
2014-08-14 11:03:08,969 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 175 in 7
counting down 957271051
2014-08-14 11:03:08,983 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 176 in 14
counting down 260602422
2014-08-14 11:03:08,991 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 177 in 8
counting down 1344752100
2014-08-14 11:03:08,996 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 178 in 5
counting down 210373011
2014-08-14 11:03:09,003 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 179 in 7
counting down 1135342237
counting down 255748372
2014-08-14 11:03:09,008 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 180 in 5
2014-08-14 11:03:09,013 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 181 in 5
counting down 1706850686
2014-08-14 11:03:09,018 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 182 in 5
counting down 958800587
2014-08-14 11:03:09,023 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 183 in 5
counting down 621084776
counting down 1494203157
2014-08-14 11:03:09,028 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 184 in 5
2014-08-14 11:03:09,034 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 185 in 6
counting down 1610501827
2014-08-14 11:03:09,039 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 186 in 5
counting down 1472601997
2014-08-14 11:03:09,046 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 187 in 6
counting down 1523401565
2014-08-14 11:03:09,052 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 188 in 6
counting down 2118032386
2014-08-14 11:03:09,058 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 189 in 6
counting down 1402690896
2014-08-14 11:03:09,063 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 190 in 5
counting down 877005529
counting down 69300391
2014-08-14 11:03:09,068 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 191 in 5
counting down 250950317
2014-08-14 11:03:09,074 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 192 in 6
2014-08-14 11:03:09,080 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 193 in 6
counting down 126717950
2014-08-14 11:03:09,086 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 194 in 6
counting down 60378388
2014-08-14 11:03:09,092 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 195 in 6
counting down 213037448
2014-08-14 11:03:09,098 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 196 in 6
counting down 737512169
2014-08-14 11:03:09,103 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 197 in 5
counting down 1089208854
2014-08-14 11:03:09,109 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 198 in 6
counting down 156501746
2014-08-14 11:03:09,115 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 199 in 6
counting down 284416161
2014-08-14 11:03:09,121 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 200 in 6
2014-08-14 11:03:09,123 INFO [com.couchbase.client.CouchbaseConnection] - Shut down Couchbase client
2014-08-14 11:03:09,125 INFO [com.couchbase.client.ViewConnection] - I/O reactor terminated
2014-08-14 11:03:09,125 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - DONE!
Disconnected from the target VM, address: '127.0.0.1:61775', transport: 'socket'

Process finished with exit code 0

Comment by Michael Nitschinger [ 14/Aug/14 ]
https://github.com/mbsimonovic/jepsen-couchbase/pull/2
Comment by Matt Ingenthron [ 18/Aug/14 ]
Milan: would be great if you could update your blog with a paragraph or so at the top clarifying that it did work so people don't have to read into the comments to find that it was a problem with the test.
Comment by Milan Simonovic [ 19/Aug/14 ]
Sure, as soon as I get back from holidays (next we latest). Ps the tracker is major pain in the a** to use on a smartphone..
Comment by Milan Simonovic [ 26/Aug/14 ]
updated, feel free to close the issue
Comment by Michael Nitschinger [ 26/Aug/14 ]
Not an issue.




[JCBC-499] Test SSL Feature with Java 2.0 SDK Created: 28/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Closed
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ssl.pcapng    

 Description   
Michael, here is the SSL Feature Test I created
https://github.com/couchbase/couchbase-java-client/pull/6/files
I know there are suff should be in core, and we can move it later

There are a couple issues in this feature tests

1. view and query test hangs
2. the ssl test I created is not able to catch exceptions such as sun.security.validator.ValidatorException, javax.net.ssl.SSLHandshakeException. etc

We can’t disable SSL from the server side after starting the server. SSL certificate automatically generated and self signed, so we can’t modify expired date.

I also test 2.5 server connect with java client, which does not work with SSL.

Attached file is wireshark packet capture fro FeatureTest with SSL. If you decode it with SSL in Wireshark, you can see the data and TLS protocol.
  



 Comments   
Comment by Wei-Li Liu [ 30/Jul/14 ]
java sdk's commit on observe relies on core
https://github.com/weilliu/couchbase-java-client/commit/6f8d672ff9a600780fd5d7d47d5ba6ce52bd083d

import com.couchbase.client.core.message.binary.ObserveRequest;
import com.couchbase.client.core.message.binary.ObserveResponse;

Currently Core can't be built on master

====================================
[root@centos-64-x64 couchbase-jvm-core]# ./gradlew install
:compileJava
Note: /root/sdkd-2.0/couchbase-jvm-core/src/main/java/com/couchbase/client/core/CouchbaseCore.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:processResources
:classes
:jar
:javadoc
java.lang.UnsupportedClassVersionError: ch/raffael/doclets/pegdown/PegdownDoclet : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at com.sun.tools.javadoc.DocletInvoker.<init>(DocletInvoker.java:96)
at com.sun.tools.javadoc.Start.setDocletInvoker(Start.java:414)
at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:213)
at com.sun.tools.javadoc.Start.begin(Start.java:162)
at com.sun.tools.javadoc.Main.execute(Main.java:59)
at com.sun.tools.javadoc.Main.main(Main.java:49)
javadoc: error - fatal error
1 error
:javadoc FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':javadoc'.
> Javadoc generation failed.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 24.87 secs
====================================================

Comment by Wei-Li Liu [ 04/Aug/14 ]
Please disregard my above error, which is triggered by inconsistent jdk version on run time and compile time.
Comment by Don Pinto [ 11/Aug/14 ]
Wei-li, can you update this ticket?
Comment by Wei-Li Liu [ 11/Aug/14 ]
@Michael looking at your most recent commit, I don't think the catching exception issue has been fixed yet. I believe you just changed the fix version to 2.0-beta.
I also have some issues build couchbase-java-client from master after I pull both sdk and the core. I knew Deepti has the same issue earlier, but not sure if it has been resolved yet.
Comment by Matt Ingenthron [ 18/Aug/14 ]
Michael: this is blocking completion of testing, can you aim to resolve it this week.
Comment by Wei-Li Liu [ 18/Aug/14 ]
After getting some help from Deepti, I am able to get dp3 build with ./gradlew publish and ./gradlew publishToMavenLocal
Integration test has been changed in dp3, so some modifications required for SSL feature tests on my end.
Comment by Wei-Li Liu [ 20/Aug/14 ]
Report bug JCBC-520
Comment by Michael Nitschinger [ 21/Aug/14 ]
So are we tracking it in JCBC-520 and closing this?
Comment by Wei-Li Liu [ 21/Aug/14 ]
okay




[JCBC-498] I can't connect to an authenticated bucket Created: 28/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Task Priority: Major
Reporter: Gareth Powell Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: MacOS X

Attachments: Text File trace.txt    

 Description   
I can't connect to a password-protected bucket with SDK 2-dp2.

"Similar" code can connect with SDK 1.4.

Here is my approximate code:

cluster = new CouchbaseCluster(couchConfig.fromHost);

Bucket bucket = cluster.openBucket(couchConfig.bucket, couchConfig.passwd).toBlocking().single();

if (couchConfig.clearDB)

bucket.flush().toBlocking().single();

where "fromHost" is set to "warburton", "bucket" is ziggrid-baseball and password is "". (Yes, that is the password on the bucket in development).

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
thanks, I'll look into it soon. Let's see if I can fix that for dp3
Comment by Gareth Powell [ 20/Aug/14 ]
I just tried this and in dp3 I can "kind of" get it to work.

The thing is, I couldn't use the code above (presumably due to API changes) and that suggests the code above might always have been invalid.

I changed:

cluster = new CouchbaseCluster(couchConfig.fromHost);

to

cluster = CouchbaseCluster.create(couchConfig.fromHost);

and

bucket.flush().toBlocking().single();

to

bucket.bucketManager().toBlocking().single().flush().toBlocking().single();
Comment by Michael Nitschinger [ 20/Aug/14 ]
Yup, your changes look good. What do you mean by kind of? Does it work with the changes?
Comment by Gareth Powell [ 20/Aug/14 ]
Yes, it works now. At the time I made the comment, I was able to connect and clear out the bucket, but nothing else appeared to be working.

The problem I'm having is the usual migration one: I have so many errors in my code from the API changing that it's hard to tell what is going wrong. It turns out that my line of execution was being sabotaged by a broken dependency that wasn't getting caught. When I was able to fix that, I was able to get to the stage of loading a design document.

I now have design documents and regular documents loading.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Perfect, thanks. Let me know if you run into more issues.




[JCBC-497] reduce in ViewQuery doesn't work Created: 27/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When querying using views, setting the reduce(false) seems to have no influence on the query and always reduces the query.

Am I missing something?

 Comments   
Comment by Michael Nitschinger [ 28/Jul/14 ]
Thanks, I'll look into it.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Works in dp3 and on master!

Make sure that if you have a view with a reduce function, set .reduce(false) on the DSL to disable it optionally.




[JCBC-496] Add unit test for JCBC-488 Created: 22/Jul/14  Updated: 28/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .next
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We need to add a unit test for the fix in http://www.couchbase.com/issues/browse/JCBC-488 just to make sure it works as expected. The appropriate change ishttp://review.couchbase.org/#/c/39429/ and has been merged already.

I'd propose a unit test which mocks out the underlying stuff and just pushes a HttpOperation a few times through the ViewConnection#addOp which should add the headers only 1 times instead of N.

 Comments   
Comment by Wei-Li Liu [ 25/Jul/14 ]
I think we have something similar to JCBC-488 test case in the scenario test framework (Service Restart)

I pull in your fix http://review.couchbase.org/#/c/39429/ and reproduce the issue by switching thread # to 200 instead of 10 (default)
The report shows that if thread is 10 works for both mc (get,set) and ht, cb (view, query). If jumps thread # to 200, view and query workload would fail
https://docs.google.com/spreadsheets/d/1sjDpJL_xuVv3kDhbnJKn5qNLmvzCi80gbdq5FkOObzQ/edit#gid=1476387871

I can make a separate unit test to make sure when there are multiple http operations, there can only be 1 header.






[JCBC-495] 1.4x doesn't build if source dir contains whitespace Created: 16/Jul/14  Updated: 04/Aug/14  Resolved: 04/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.3
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Minor
Reporter: David Haikney Assignee: David Haikney
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
After running a git clone; git checkout of 1.4.3 I was unable to build on my local machine. "ant package" yielded the following:

 [exec] /Users/dhaikney/Documents/Google Drive/dev/couchbase-java-client/src/scripts/write-version-info.sh: line 47: ${changesfile}: ambiguous redirect

Some of the variables in the write-version-info.sh script need surrounding quotes to allow for the whitespace.

 Comments   
Comment by David Haikney [ 16/Jul/14 ]
http://review.couchbase.org/39443




[JCBC-494] Support touch and unlock Created: 16/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-492] Support get + lock and get + touch Created: 15/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-491] Support counter Created: 15/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-490] Log JSON body when failed to parse valid body Created: 14/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This change does not fix an error itself, but makes it much easier to discover what valid JSON gets sent over where we are failing to handle it.




[JCBC-489] Support Replica Read Created: 10/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-488] Getting Failed to Access the View Exception Created: 08/Jul/14  Updated: 07/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0, 1.4.1
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: meet_ravip Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Relates to

 Description   
Caused by: java.lang.RuntimeException: Failed to access the view
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:787)


Caused by: java.util.concurrent.ExecutionException: OperationException: GENERAL
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:98)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:780)
        ... 9 more
Caused by: OperationException: GENERAL
        at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:74)
        at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
        at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
        at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
        at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)


 Comments   
Comment by Michael Nitschinger [ 08/Jul/14 ]
Hi,

can you please give us more information on when you are seeing the exception (what code are you executing, whats going on at the server - nothing, rebalance, failover,..) Thanks!
Comment by meet_ravip [ 08/Jul/14 ]
Hi,
  We use couchbase 1.4 client and its dependent binaries and below is the code snippet. The failure happens in client.query method. We are running 200 concurrent request to perform the below operation. When couchbase running for more than few and if we start this test it fails, after restarting couchbase when this issue happens and restart the test then it works. Our's 3 node cluster setup. At that time failure there is no rebalance or failover. Lets know if you need any further details.
                        

CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
cfb.setOpTimeout(opTimeOut); //100000
cfb.setOpQueueMaxBlockTime(queueMaxBlockTime);30000
cfb.setTimeoutExceptionThreshold(timeoutExceptionThreshold);//998
cfb.setReadBufferSize(readBufSize);//16384
cfb.setShouldOptimize(shouldOptimize);//false
cfb.setMaxReconnectDelay(maxReconnectDelay);//30
cfb.setObsPollInterval(obsPollInterval);//10
cfb.setObsTimeout(obsTimeOut);//5
cfb.setViewTimeout(viewTimeout);//300
cfb.setViewWorkerSize(viewWorkerSize);//1
cfb.setViewConnsPerNode(viewConnectionsPerNode);//100

CouchbaseClient client = new CouchbaseClient(cfb.buildCouchbaseConnection(
hosts, bucketName, userName, password));


Query query = new Query();
query.setReduce(false);
// query.setIncludeDocs(true);
query.setIncludeDocs(false);
query.setStale(Stale.FALSE);


View view = client.getView(designDocumentName, viewName);
if (view != null) {
ViewResponse result = client.query(view, query);
if (result != null) {
List<String> keys = new ArrayList<String>();
for (ViewRow row : result) {
if (row.getKey() != null && !row.getKey().isEmpty()) {

keys.add(row.getKey());
}
}
if (keys.size() > 0) {

documents = client.getBulk(keys);
}
}
}

View Definition:-

function (doc,meta) { emit(meta.id,null);
Comment by Vladislav Koroteev [ 08/Jul/14 ]
I have same issue. I noticed that this exception occured when I do request to view and view is indexing.
Comment by RICHARDS PETER [ 09/Jul/14 ]
In our use case, data update in the couchbase bucket happens only once in a day. During the remaining day, all our clients only read the data from these buckets and still we face this issue. Could you please tell us whether it is possible to control indexing of the view?
Comment by Michael Nitschinger [ 09/Jul/14 ]
Thanks folks,

could you also tell me which server versions you are using? the Error: General points to a 500 response from the server, for whatever reason.
Comment by RICHARDS PETER [ 09/Jul/14 ]
Hi Michael,
We are using 2.1.1 community edition (build-764).
Comment by RICHARDS PETER [ 09/Jul/14 ]
Hi Michael,
Could you also tell me the server log files that should be analyzed to find more details about this error?
Comment by Michael Nitschinger [ 09/Jul/14 ]
Peter,

there is a cbcollect_info command that you can use to aggregate this info, but I would be interested in something else if you can take the time: enable FINEST logging (detailed info here http://nitschinger.at/Logging-with-the-Couchbase-Java-Client) and can you share that log? in there we should have more info from the actual server response and see whats going on.

Also, you could set a breakpoint at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:74) and print the original exception https://github.com/couchbase/couchbase-java-client/blob/release14/src/main/java/com/couchbase/client/protocol/views/ViewOperationImpl.java#L74

maybe we should rethrow that.
Comment by Vladislav Koroteev [ 09/Jul/14 ]
I tried reproduce "java.lang.RuntimeException: Failed to access the view", but I can't. I do workload on Couchbase, so I can observer permanent indexing at web-interface. From JavaSDK I do query with stale=false. Earlier I get "java.lang.RuntimeException: Failed to access the view", now I have

java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:781)
at my.pack.MainView.main(MainView.java:29)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
... 2 more

I tried it on Couchbase Server 2.2 and Couchbase Server 2.5.1 EE, on Java SDK 1.3.2, 1.4.1, 1.4.3.

Maybe this exception occures while view is compacting, not indexing?
Comment by Michael Nitschinger [ 14/Jul/14 ]
I have a change pending that will at least print out the unparsable/unexpected JSON body so it will be easier to debug: http://review.couchbase.org/#/c/39347/
Comment by RICHARDS PETER [ 15/Jul/14 ]
Hi Michael,

Sorry for the delay in giving more details. A system upgrade at our side delayed further testing of this issue. I downloaded the source code of the couchbase client and added some log statements before the new exception is set. The code looks as follows:

************************************************************************************
  @Override
  public void handleResponse(HttpResponse response) {
    String json = getEntityString(response);
    int errorcode = response.getStatusLine().getStatusCode();
    try {
      OperationStatus status = parseViewForStatus(json, errorcode);
      ViewResponse vr = null;
      if (status.isSuccess()) {
        vr = parseResult(json);
      } else {
        parseError(json, errorcode);
      }

      ((ViewCallback) callback).gotData(vr);
      callback.receivedStatus(status);
    } catch (ParseException e) {
System.out.println("ViewOperationImpl handleResponse()"+e.getMessage());
e.printStackTrace();
      LOGGER.log(Level.WARNING,"ViewOperationImpl handleResponse() "+e.getMessage(),e);
setException(new OperationException(OperationErrorType.GENERAL,
          "Error parsing JSON"));
    }
    callback.complete();
  }

************************************************************************************

I was able to reproduce the issue with the new jar file packaged with the above code change. Please find the logs in our application:

2014-07-15 12:11:28 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-15 12:11:28 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-15 12:11:28 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)

Please let me know what could be going wrong. The issue does not occur always. It mostly happens after some restarts of our application and only a few clients are getting this issue.

Thanks,
Richards Peter.
Comment by Michael Nitschinger [ 15/Jul/14 ]
Hey, could you do me a favor and print both the HttpResponse and or the raw content body? from the log message I cannot see what actually got returned from the server only that we were unable to parse it.
Comment by RICHARDS PETER [ 16/Jul/14 ]
Hi Michael,

I added more print statements in the catch block to capture the request and response. Please find the modified code and the logs:

 public void handleResponse(HttpResponse response) {
   //retained the same code till here.
    } catch (ParseException e) {
System.out.println("ViewOperationImpl handleResponse()"+e.getMessage());
e.printStackTrace();
System.out.println("ViewOperationImpl handleResponse() HttpRequest: "+getRequest().toString());
System.out.println("ViewOperationImpl handleResponse() HttpResponse: "+response.toString()+" json:"+json+" errorcode:"+ errorcode);
      LOGGER.log(Level.WARNING,"ViewOperationImpl handleResponse() "+e.getMessage(),e);
setException(new OperationException(OperationErrorType.GENERAL,
          "Error parsing JSON"));
    }
    callback.complete();
  }

The issue was reproduced today morning. Please find our application log:

2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-16 10:36:50 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)


2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-16 10:36:50 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)

2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse() HttpRequest: GET /COMMON_DIMENSION/_design/COMMON_DIMENSION/_view/COMMON_DIMENSION?reduce=false&stale=false [Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Connection: Keep-Alive, User-Agent: JCBC/1.2, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q
2014-07-16 10:36:50 STDIO [INFO] 09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09
2014-07-16 10:36:50 STDIO [INFO] NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NT
2014-07-16 10:36:50 STDIO [INFO] U9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9
2014-07-16 10:36:50 STDIO [INFO] OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092]


2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse() HttpResponse: HTTP/1.1 400 Bad Request [Server: MochiWeb/1.0 (Any of you quaids got a smint?), Date: Wed, 16 Jul 2014 05:06:50 GMT, Content-Length: 0] json: errorcode:400
Comment by Michael Nitschinger [ 16/Jul/14 ]
Hi,

thanks for reporting this. I'm not exactly sure whats going on here, could it be that all those authorization headers are part of _one request_?

Can you clarify if you are using the default bucket or something named differently?
I think this is a bug, but I'm not entirely sure if this is the same as the others reported here, let me get a fix up for this since this seems to be trivial.
Comment by RICHARDS PETER [ 16/Jul/14 ]
We have created a new bucket - COMMON_DIMENSION and this bucket has a view also with the same name. The view definition and code to retrieve the data is already shared by meet_ravip.
Comment by Michael Nitschinger [ 16/Jul/14 ]
Thanks, the good news is that I have a trivial bugfix for your case I think. Currently we retry a request if it fails for some reason against a node, but the code just adds the headers again and I guess the server just rejects it at some point. So I'll add checks around the add to make sure they are only added once.
Comment by Michael Nitschinger [ 16/Jul/14 ]
See: http://review.couchbase.org/#/c/39429/

I think this will get into 1.4.4, but you can patch it on your own if you can't wait for it. If you do, let me know if it works
Comment by RICHARDS PETER [ 16/Jul/14 ]
Thanks Michael,
That is really a good news. Please share the details of the fix when it is available. We will test the fix in our setup and update you about our findings.
Comment by Michael Nitschinger [ 16/Jul/14 ]
1.4.4 is scheduled for first week of august, but I'll merge it into master much sooner (this week I hope) once it goes through review. If you know how to build it from github (if not I can help you with) you can play around with it sooner.
Comment by RICHARDS PETER [ 16/Jul/14 ]
Thanks Michael.
I am about to test the aforementioned patch on my setup. I have applied the changes to the 1.4 source code which we downloaded a few days back. I will update you about our observation. Since this issue cannot be reproduced very quickly, my update on this issue may be slightly delayed (24 hours minimum).
Comment by RICHARDS PETER [ 17/Jul/14 ]
Hi Michael,
I would like to clarify one more doubt related to this comment:
"Currently we retry a request if it fails for some reason against a node, but the code just adds the headers again and I guess the server just rejects it at some point. "
We would like to minimize such failures. Is it possible to find out the reason why the request failed against a node? Is it possible to capture such cases in our log by modifying any class? Could you please guide us on the same?
Comment by Michael Nitschinger [ 17/Jul/14 ]
Hi,

normally you don't need to be concerned about that. There are various situations as part of the regular operations (like adding a node) where there is a limited timespan where the node is added, but does not contain any partitions yet. We get a response from the server and retry behind the scenes. Also with failovers where the config is not updated the node returns an error and we retry.

If we don't get to a good result we will try until the op times out with the given timeout, so you should be good on that front. Of course, the behavior reported here was because of a bug.
Comment by RICHARDS PETER [ 17/Jul/14 ]
Thanks Michael,
We normally get this error after some restart of our application. Till now we were not able to reproduce this issue. However we got a new error after the latest restart of our application. The stacktrace is as follows:

Caused by: com.couchbase.client.vbucket.ConfigurationException: Could not fetch a valid Bucket configuration.
at com.couchbase.client.vbucket.provider.BucketConfigurationProvider.bootstrap(BucketConfigurationProvider.java:123)
at com.couchbase.client.vbucket.provider.BucketConfigurationProvider.getConfig(BucketConfigurationProvider.java:373)
at com.couchbase.client.CouchbaseConnectionFactory.getVBucketConfig(CouchbaseConnectionFactory.java:317)
at com.couchbase.client.CouchbaseClient.(CouchbaseClient.java:258)

All that we did was an application restart. There were no other changes in the environment. We could see all the buckets on the couchbase nodes when this error was produced. The buckets also had the same items as before. During the restart there will be around 200 threads accessing the same data. I would like to know the possible scenarios which can lead to this error. Would you like me to open a new issue for this?
Comment by Michael Nitschinger [ 17/Jul/14 ]
Yes please, and if you could provide debug level logging (or finest even) would be good, thanks!




[JCBC-487] com.couchbase.client.protocol.views.Qery key construct Created: 07/Jul/14  Updated: 15/Jul/14  Resolved: 15/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 1.4.1, 1.4.3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: reinerRubin Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Hello.
There is a little bug in query.setKey. When

String login = "0987654321";
Query query = new Query();
query.setKey(login);

query will be with ?key=987654321 not 0987654321.

It's happen in query.quote()
(https://github.com/couchbase/couchbase-java-client/blob/1.4.3/src/main/java/com/couchbase/client/protocol/views/Query.java#L292)


Also in version 1.3.0 in similar situation (key = "0987654321") key will be key=0987654321 and in response
java.util.concurrent.ExecutionException: OperationException: SERVER: bad_request Reason: invalid UTF-8 JSON: {{error,garbage_after_value},\"0987654321\"}

// Excuse my bad English

 Comments   
Comment by Michael Nitschinger [ 07/Jul/14 ]
You should use ComplexKey instead of the string directly.

Try:

query.setKey(ComplexKey.of("0987654321");
Comment by reinerRubin [ 07/Jul/14 ]
>You should use ComplexKey instead of the string directly.
Thanks. It's work.
But I don't sure about setKey(String) behavior.
Comment by Michael Nitschinger [ 15/Jul/14 ]
Yeah, with the string its not so straightforward, thats why we added the complex key. for now please stick to it, it will be easier with 2.0
Comment by reinerRubin [ 15/Jul/14 ]
Ok. Thanks!




[JCBC-485] JsonObject and all its similars Created: 06/Jul/14  Updated: 24/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0.0
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: json
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Why should I work with JsonObject, JsonArray and all of that? Why can't I just send a String and you guys do with it what ever you want? Or at least give us a utility to convert our Pojo to your JsonObject?

If there is such utility I take it back :)

 Comments   
Comment by Michael Nitschinger [ 07/Jul/14 ]
Hi,

yes you are right, and we will provide that functionality as well once we get to feature completeness on 2.0. There has already been a user contribution to a converter which will act like the one on 1.4 so you can then pass in a string and it will be stored. So you get the both of two worlds.

Does that work for you?
Comment by Michael Nitschinger [ 21/Aug/14 ]
We have implemented a LegacyDocument where you can pass in a string and it works like the old one. We probably add more documents (string,...) directly so you can have more fun with it.

I'll close this one for now since it works with beta/dp3 already using the legacy codec and open more issues for specific transcoders/documents.
Comment by Oded Hassidi [ 24/Aug/14 ]
Thanks




[JCBC-484] Publish docs for Java July 2014 release Created: 01/Jul/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Closed
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.4.3
Fix Version/s: 1.4.3
Security Level: Public

Type: Task Priority: Critical
Reporter: Amy Kurtzman Assignee: Amy Kurtzman
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 1h
Time Spent: Not Specified
Original Estimate: 1h


 Description   
Publish updated documentation, release notes, and javadocs.




[JCBC-483] Add INFO level log of client version to startup Created: 27/Jun/14  Updated: 05/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.2
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In CouchbaseClient <init> we log details from the CouchbaseConnectionFactory. It would also be useful to INFO level log the client version and the spymemcached version.

 Comments   
Comment by Matt Ingenthron [ 27/Jun/14 ]
Since it's such a minor change, setting it to 1.4.3. I'll defer to you on whether it should be pushed back or not.
Comment by Michael Nitschinger [ 30/Jun/14 ]
It's a little harder to do because there is the BuildInfo, but we need to make sure its properly in path and distributed with our jars. I'm not sure this is done right now but I could be wrong.

If we could defer it to 1.4.4 then it would be better to release 1.4.3 tomorrow.




[JCBC-482] Select proper replica node for getsFromReplica Created: 27/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-481] Integration Tests hang up after 85% progress Created: 25/Jun/14  Updated: 15/Sep/14  Resolved: 15/Sep/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
[root@pomelo-11017 couchbase-java-client]# ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava UP-TO-DATE
:processIntegrationResources UP-TO-DATE
:integrationClasses UP-TO-DATE
> Building 85% > :integrationTest

 Comments   
Comment by Wei-Li Liu [ 18/Aug/14 ]
The process hangs at 85% due to view query test.
Just runs 2.0.0-dp3 integration test against the new beta2 build, and all the tests completes.
Comment by Deepti Dawar [ 21/Aug/14 ]
Following error appears with latest DP3 -

[root@pomelo-11017 couchbase-java-client]# ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processIntegrationResources
:integrationClasses
:integrationTest

com.couchbase.client.java.BinaryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.BinaryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.DesignDocumentTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.DesignDocumentTest > classMethod FAILED
    java.util.NoSuchElementException

4 tests completed, 4 failed
:integrationTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integrationTest'.
> There were failing tests. See the report at: file:///root/couchbase/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 22.09 secs
Comment by Michael Nitschinger [ 21/Aug/14 ]
Does it also happen on master?
Comment by Wei-Li Liu [ 21/Aug/14 ]
I test it on master and only see JCBC-479




[JCBC-480] replica read with NOT_EXISTS never completes Created: 25/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-479] ComparisonFailure in running unit tests for 2.0 client Created: 24/Jun/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 2.0.0
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test

com.couchbase.client.java.document.json.JsonObjectTest > shouldExportMix FAILED
    org.junit.ComparisonFailure at JsonObjectTest.java:27

com.couchbase.client.java.query.ViewQueryTest > shouldHandleEndKey FAILED
    org.junit.ComparisonFailure at ViewQueryTest.java:178

77 tests completed, 2 failed
:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///root/couchbase/sdkd-2.0/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED


 Comments   
Comment by Wei-Li Liu [ 15/Jul/14 ]
I am getting different failure running unit test

===================================================================================
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test

com.couchbase.client.java.convert.JacksonJsonConverterTest > shouldEncodeSimpleJsonObject FAILED
    org.junit.ComparisonFailure at JacksonJsonConverterTest.java:70

com.couchbase.client.java.convert.JacksonJsonConverterTest > shouldEncodeMixedNestedJsonValues FAILED
    org.junit.ComparisonFailure at JacksonJsonConverterTest.java:177

77 tests completed, 2 failed
:test FAILED

FAILURE: Build failed with an exception.
Comment by Wei-Li Liu [ 15/Jul/14 ]
Failure running integration tests


============================================================================
wei-lis-mbp:couchbase-java-client wei-li$ ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava UP-TO-DATE
:processIntegrationResources UP-TO-DATE
:integrationClasses UP-TO-DATE
:integrationTest

com.couchbase.client.java.BinaryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.BinaryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.QueryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.QueryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.ViewTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.ViewTest > classMethod FAILED
    java.util.NoSuchElementException

6 tests completed, 6 failed
:integrationTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integrationTest'.
> There were failing tests. See the report at: file:///Users/wei-li/repo/couchbase/sdkd-2.0/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Comment by Wei-Li Liu [ 15/Jul/14 ]
Disregard the above integration test failure. It requires to create "default" bucket and set no password to run.
Comment by Wei-Li Liu [ 18/Aug/14 ]
Running 2.0-dp3 feature test against 1118 server, and all of the unit test and integration tests passed. I am resolving this ticket.
Comment by Deepti Dawar [ 19/Aug/14 ]
Hi Michael, Please close this when you push the fix for it.

Thanks !
Comment by Michael Nitschinger [ 19/Aug/14 ]
Should be fixed in master, please reopen if not.
Comment by Wei-Li Liu [ 20/Aug/14 ]
I run int test on master today and there is 1 failure on shouldUpsertLegacyObjectDocument

com.couchbase.client.java.BinaryTest STANDARD_ERROR
    log4j:WARN No appenders could be found for logger (com.couchbase.client.core.logging.CouchbaseLoggerFactory).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument FAILED
    java.lang.NullPointerException
        at com.couchbase.client.java.BinaryTest.shouldUpsertLegacyObjectDocument(BinaryTest.java:295)
Gradle Worker 1 finished executing tests.

28 tests completed, 1 failed
Comment by Michael Nitschinger [ 21/Aug/14 ]
Hm there might be an issue with the new legacy document, a different test is failing for me.

Is there any other info in the logs what's going on?
Comment by Deepti Dawar [ 21/Aug/14 ]
The unit tests now run fine.
Tried with the latest of DP3.
Comment by Wei-Li Liu [ 21/Aug/14 ]
stacktrace for legacy document test

11:40:54.879 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument STARTED
11:40:54.918 [DEBUG] [TestEventLogger]
11:40:54.918 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument FAILED
11:40:54.918 [DEBUG] [TestEventLogger] java.lang.NullPointerException
11:40:54.918 [DEBUG] [TestEventLogger] at com.couchbase.client.java.BinaryTest.shouldUpsertLegacyObjectDocument(BinaryTest.java:295)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.919 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
11:40:54.921 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.922 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
11:40:54.922 [DEBUG] [TestEventLogger] at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.923 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
11:40:54.923 [DEBUG] [TestEventLogger] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
11:40:54.923 [DEBUG] [TestEventLogger] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
11:40:54.923 [DEBUG] [TestEventLogger] at java.lang.Thread.run(Thread.java:745)
11:40:54.925 [DEBUG] [TestEventLogger]
11:40:54.925 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldInsertAndGet STARTED
11:40:54.925 [DEBUG] [TestEventLogger]
11:40:54.925 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldInsertAndGet PASSED
11:40:57.110 [QUIET] [system.out] 11:40:57.110 [
11:40:57.110 [DEBUG] [TestEventLogger]
11:40:57.110 [QUIET] [system.out] DEBUG] [org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor] Executing test class com.couchbase.client.java.DesignDocumentTest
Comment by Wei-Li Liu [ 21/Aug/14 ]
This exception is caused by integration test, which is somewhat a duplicate of JCBC-481.
unit test works fine for me too.




[JCBC-478] Docs: Deferred view deployment instruction does not work Created: 17/Feb/14  Updated: 19/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.0
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Minor
Reporter: Don Stacy Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Section: http://docs.couchbase.com/couchbase-sdk-java-1.3/#preview-the-application
Area: Text under step 2
Issues: We say that we will deploy the views to production later. However, the app will not work unless the views are in production. We should include the steps here or at least tell the reader to publish while pointing them to the View documentation.




[JCBC-477] Pass view timeout down to get bulk when docs used Created: 23/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-476] Rework now misleading WARN message. Created: 23/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-475] Harden DefaultConfigurationProvider on restart and shutdown. Created: 17/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency




[JCBC-474] Should be able to specify Transcoder to use when querying a view Created: 10/Jun/14  Updated: 16/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.1
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Minor
Reporter: raycardillo Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
All of the other methods in the interface support providing a Transcoder on each call and not necessarily relying on the default Transcoder configured in the connection builder. However, there is no way to do this when querying a view. This really only becomes an issue in combination with the "include documents" feature. When you're working with a document that has been Transcoded it will fail when you call getDocument() on the results unless you register a default Transcoder. It would be more consistent if we could pass a Transcoder to the getDocument() call (or configure it for the Query object)... and it would be even nicer if we could benefit from the duck typing logic to get the document as a JSON String if it's been identified by Couchbase as json.




[JCBC-473] Flush support Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-472] Implement support for snappy compression where supported Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-471] Implement HELLO command Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-470] Implement Carrier-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-469] Implement HTTP-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-468] Implement EBNF-based N1QL DSL Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-467] Java client not aware of failed over node under certain circumstances Created: 10/Jun/14  Updated: 23/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: .next
Security Level: Public

Type: Task Priority: Critical
Reporter: Robert Waite Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File anotherMultithreadTest.java     Java Source File CouchbaseClientTesterFastAndThreaded.java     Java Source File CouchbaseClientTester.java     Java Source File CouchbaseClientTesterNewer.java     Text File logAfterSeemingFreeze.txt     Text File threadDumpDuringFreezeSunJVM.txt     Text File threadDumpDuringFreeze.txt    

 Description   
This is similar to http://www.couchbase.com/communities/q-and-a/java-client-not-aware-about...

This is using the 1.4.2 java client.

If I failover from the admin console while the node is still up... it does not reproduce. I found the best way to simulate the node suddenly becoming unresponsive is to use IP tables to block all traffic except SSH on port 22 like so:

To block a node:
iptables -A INPUT -p tcp --dport 22 -j ACCEPT; iptables -A INPUT -j DROP

To unblock a node:
iptables -F

It also seems to not reproduce if I remove the nested try/catch (i.e. don't try to read from the replica).

Failover seems to not be instantaneous... it takes 1-2 minutes with my hardware and setup. The following steps can be seen with the code below (might be reproducible with less steps but this seems consistent):

1) Create a two node cluster with 1 level of replication
2) Set the code below with the proper host names, bucket name and password
3) Run the code and you will see "Got From Master" 5 times
4) It will then pause and ask you to block traffic from the master node
5) Look at the admin console to see which node is the master for the key
6) Block the master node with iptables and then hit 'Enter'
7) Go back to the output and it will then output "Got From Replica" 10 times
8) It will then pause and ask for you to go to the admin console (on the replica node)
9) Wait until the master node is marked as "Down"
10) Once it is marked as "Down", fail over the master node
11) Go back to the console and hit 'Enter'

At this point the console should continue printing "Got From Replica". If you look at the admin console the replica node still has 0 items active and 1 item replicated. After 1-3 minutes it should suddenly say 1 item is active and 0 items are replicated (failover seems delayed). You will also notice at the same time that an exception started showing up in the console.

Expected: Once the node is fully failed over, it should no longer need to read from the replica and should read from the promoted master

Observed: It doesn't seem to be able to read from the master or the replica. It appears that the client is not marking the promoted replica as the new master.

Questions:

1) What is going on during the failover? I would have thought that failover would have been very fast and not take 1-5 minutes. Especially since I only have one item in the store
2) Anyone know of a workaround? If I catch the exception and rebuild the client... it works. But this would be horrible since the client is accessed by multiple threads.


 Comments   
Comment by Robert Waite [ 10/Jun/14 ]
Attached testing app. (CouchbaseClientTester.java)
Comment by Michael Nitschinger [ 10/Jun/14 ]
Thanks for the report, looking into it. Will provide an update as soon as I find something.
Comment by Michael Nitschinger [ 11/Jun/14 ]
Which server version are you using?
Comment by Michael Nitschinger [ 11/Jun/14 ]
If its 2.5.0 or 2.5.1, could you do me a favor and try setting "System.setProperty("cbclient.disableCarrierBootstrap", "true");" before the CouchbaseClient initialisation and report if it changes things? If its 2.2 nevermind.
Comment by Robert Waite [ 11/Jun/14 ]
It is 2.2.0. I can install 2.5.1 and give it a shot though. I had actually tried the disableCarrierBootstrap option but didn't realize that it wasn't available against the 2.2.0 server.
Comment by Robert Waite [ 11/Jun/14 ]
I upgraded the two nodes to 2.5.1 and added the disableCarrierBootstrap property before client creation. I was still able to repro the issue. There are some new details though:

1) The test will not always reproduce (even against the 2.2.0 server). Sometimes when you click failover from the admin console you will see the following message:

2014-06-11 10:01:35.889 INFO com.couchbase.client.CouchbaseConnectionFactory: Replacing current streaming node list [http://node1:8091/pools, http://node2:8091/pools] with [http://node1:8091/pools]
2014-06-11 10:01:35.894 INFO com.couchbase.client.CouchbaseConnection: Scheduling Node node3/10.1.1.10:11210for shutdown.

When this happens I normally see one more replica get, then it goes to the master and functions like I imagine it should. When testing against 2.5.1 this seems to happen 1 in 3 times. I think it was much less common with 2.2.0... maybe 1 in 8 times. My guess is that this could be related to how quickly failover seems to happen now in 2.5.1. Before... with 2.2.0 the pattern was wait about 1 minute to see the node marked as down, click failover and then see the active item show up on the other node after maybe 30 seconds at which point the error would show. With 2.5.1 the pattern is wait about 1 minute for the node to be marked as down, click failover and then immediately see the item show up on the other node as active and then the error will show.

2) Adding the disableCarrierBootstrap property did not seem to have any effect. I tried with and without maybe 5 times each. Both would show the error about 2 in 3 tests.

3) I altered the test to add the system property and to also continue trying replica gets continuously instead of pausing and waiting for the tester to click failover. Now steps 7+ from the original description are different. Once you complete 6 you should look at the admin console and wait for it to mark the node as down. Once it does, you click failover. If the error occurs then you will see exceptions. If the error does not happen you will see the INFO messages about replacing the current streaming node list and then it will start saying it is reading from the master instead of the replica (which is the desired behavior).
Comment by Robert Waite [ 11/Jun/14 ]
Attaching new version of testing app as CouchbaseClientTesterNewer.java
Comment by Michael Nitschinger [ 11/Jun/14 ]
Thanks very much for the additional info, I'll try to dig in this week and see what I can come up with. I'll let you know if I need anything else.

Again, very much appreciated!
Comment by Robert Waite [ 11/Jun/14 ]
Oh no! When I was testing the property change my POM was pointed at the 1.3.2 client instead of the 1.4.2 client. I have done three tests so far after adjusting the POM and now it seems to recover (I see the exception once but then it starts going to the master).

With this property change I see "Carrier config not available, bootstrapped through HTTP." in the log. Does this mean that the HTTP bootstrapping is not available from servers prior to 2.5.0?
Comment by Michael Nitschinger [ 11/Jun/14 ]
No thats good. It means that carrier config is not available prior to 2.5.0, so 2.2 will use HTTP. the 1.4 SDKs have a carrier -> http fallback, but with the sys property you could disable it on 2.5 as well.
Comment by Michael Nitschinger [ 11/Jun/14 ]
So with 1.4.2 it works?
Comment by Robert Waite [ 11/Jun/14 ]
1.4.2 client definitely failed against the 2.2.0 server. Maybe something about the faster failover in 2.5.1 changed what is happening? Let me reinstall 2.2.0 and confirm.
Comment by Robert Waite [ 11/Jun/14 ]
Confirmed that 1.4.2 client still fails against 2.2.0 server. When I click failover, instead of immediately seeing the data available on the promoted master... there is the 1-5 minute delay for the failover to actually complete. Then I see exceptions (not getting from replica or master).

However, with the disableCarrierBootstrap property set... after throwing the exception for 2-3 minutes it seems to recover. I am pretty sure it did not recover without that property set but I can validate that as well.

This likely is workable as a fix on my end. Do you know why failover is slow on 2.2.0 but seems to be so fast on 2.5.1?

Thanks again for your quick replies.
Comment by Robert Waite [ 11/Jun/14 ]
Hmm... even without the property set on 1.4.2 client against 2.2.0 server it seems to recover. I am wondering if this is related to queued up requests during the failover delaying or potentially blocking the client refreshing its node list.

For example... I get errors like this sometimes:

2014-06-11 11:48:53.284 WARN com.couchbase.client.CouchbaseConnection: Cancelling operation Cmd: 0 Opaque: 2682 Key: 27d528c1-0716-4f9d-98ef-ccfeaeee1b83because it has been retried (cloned) more than 100times.

I feel like the fullblown app I am testing with does not recover. Perhaps it is because it is more heavily threaded. I will try to work on a new testing app that 1) maybe reduces the timeout so that more failed requests can happen before failover is complete and maybe 2) try multiple threads all making requests against the client.

I am wondering if the problem is exacerbated in 2.2.0 because of the long failover and that perhaps there is a way to create it in 2.5.1. I am also wondering if there are cases that don't recover... because that is what originally made me dig into this.

I'll play with it some more and let you know.
Comment by Robert Waite [ 11/Jun/14 ]
Played with decreasing the timeout from the default of 2.5 seconds to 200 milliseconds. After maybe 3 or 4 runs it would always recover (after 1-3 minutes). I ran the gets with 6 threads and saw many new types of errors and it does not seem to recover (this is all against 2.2.0... I can try 2.5.1 a little later).

When I block traffic to the master node... it starts getting from replica... and then after a little while I see these sorts of errors:

Got From Replica
Got From Replica
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
Got From Replica
Got From Replica
Got From Replica
2014-06-11 12:50:04.942 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.942 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.

At this point it is still getting data from replicas... but the warning starts showing up.

Once I kick off the failover, I see the exception start showing up (not able to get from replica or master) and finally some log messages show up about having trouble connecting happens and everything halts. The application is still running but nothing is happening... no errors. I will leave it for the next hour to see if it does somehow recover... but it is odd that timeouts no longer are spewing to the screen.

I will attach the threaded testing app and the log output when everything seems to halt.
Comment by Robert Waite [ 11/Jun/14 ]
Attaching quick timeout and threaded version of testing app. (CouchbaseClientTesterFastAndThreaded.java)
Comment by Robert Waite [ 11/Jun/14 ]
Attaching log file showing what leads up to the client seeming to lock up and no longer function. The last line in there is all that is on the screen but the java process is still alive. (logAfterSeemingFreeze.txt)
Comment by Michael Nitschinger [ 13/Jun/14 ]
Thanks for the detailed infos, I have a few more questions:

- Are you using views? Could be related to http://www.couchbase.com/issues/browse/MB-11146
- Can you do a thread dump after the observed lockup? maybe we can look in there and see whats going on.

Also, when you do a failover and all that stuff, you can check on the server logs and observe the timings where it says it started failover and once its done. Maybe this correlates with the info on the client side, I saw some other tickets (some internal support) where there were also longer failover pauses reported against 2.2 (from the server side). Maybe this helps us triage it further.

Just to clarify, against 2.5.X it seems to work fine and the observed behavior is only agains 2.2.?
Comment by Robert Waite [ 13/Jun/14 ]
We are not using views... just plain key/value pair lookups (although we are letting the client be responsible for serializing the objects... not serializing ourselves and passing strings).

I can do a thread dump.

The long failover pause only happened with 2.2.0... not 2.5.1. My guess was that it was related to MB-8039. Our installation of 2.2.0 is pretty vanilla (512MB per node limit on bucket... 1 level of replication... 2 nodes). I am pretty sure you will see the same behavior with the test code.

2.5.1 seemed to recover very quickly with property file change. My guess was that this was because the failover did not take 1-5 minutes to happen so less operations were outstanding client-side. This was just the test for recovery after failover. The issue where things locked up with the multi-threaded test was completely different... I have not tested the multi-threaded test against 2.5.1 yet.
Comment by Robert Waite [ 13/Jun/14 ]
Attached thread dump output.
Comment by Robert Waite [ 13/Jun/14 ]
Sorry... realized there were some actual hostnames in the threaddump... reattaching. (threadDumpDuringFreeze.txt)
Comment by Robert Waite [ 13/Jun/14 ]
Also attaching from Sun JVM in case there was a question about the other one being run in OpenJDK. (threadDumpDuringFreezeSunJVM.txt)
Comment by Robert Waite [ 13/Jun/14 ]
I switched the server back to 2.5.1 and in a number of tests... it did not seem to get into the frozen behavior in the client. Typically with the single thread test.. when I would click failover in 2.5.1 it would be less than half a second until the client seemed to pick up on the fact that there was a failover. With 2.2.0... it seemed to take 1-3 minutes (sometimes 5 minutes) to realize... but it would indeed failover.

It seems that the property being set had nothing to do with the fix... the only reason I thought it did was because of the issue where my POM was actually pointed to the 1.3.2 client instead of the 1.4.2 client.

So... 2.2.0 seems to have a number of issues with the various tests I have run... 2.5.1 seems resilient. It is possible that the issue is still in there though and is just harder to hit. I think the main reason 2.2.0 sees the issue so much more easily is related to the slow failover. Also... I think a unique feature of this test is that I am using replica gets and not just plain gets. I think at some point I tested commenting out the replica get and did not see the issue (but of course had no availability during that time).
Comment by Robert Waite [ 18/Jun/14 ]
Has anyone had a chance to run this? Even with 2.5.1 server and 1.4.2 client I see some odd behavior. When I cut the connection to the master node and see the replica gets... I will see a new error after maybe 20-40 seconds:

2014-06-18 11:57:28.711 WARN com.couchbase.client.CouchbaseConnection: sun.nio.ch.SelectionKeyImpl@5328f6ee exceeded continuous timeout threshold

Once this happens the data is no longer pulled from the replica and I get exceptions about timing out from any replica. This only happens when fully loaded with the 6 threads. If I back it down to 1 thread it doesn't seem to happen and didn't seem to happen at least a few times with 2 threads.

Might not be as high priority with the 2+ client coming out but I think similar tests should be run against it.
Comment by Robert Waite [ 18/Jun/14 ]
Also... I just recreated the freezing behavior in 2.5.1 server against the 1.4.2 client. I blocked a node with iptables and then just unblocked it without failing over. The client code hung. I have the thread dump but it looks pretty much the same as the others. Let me know if you want that specific dump.
Comment by Robert Waite [ 18/Jun/14 ]
I realize at this point that this ticket has ballooned to too many potential individual issues. It probably should have some subtasks created.

This is with 2.5.1 server against 1.4.2 client.

I created a new test that immediately starts performing gets with 8 threads. I then cut off a node with iptables. After about 20-40 seconds it will throw errors. If I trim it down to 1 thread I only see one error pop occaisionally, but in general it starts getting from the replica. If I bring the node back (without failover) it will start pulling from the master again after 20-60 seconds.

With this case I also added the disableCarrierBootstrap property back in and instead of reading from replicas after cutting the node off, it immediately throws errors. If I bring the node back it seems to stop processing.
Comment by Robert Waite [ 18/Jun/14 ]
Attaching newest test that was run against 2.5.1 server with 1.4.2 client. (anotherMultithreadTest.java)
Comment by Michael Nitschinger [ 23/Jun/14 ]
Thanks for your efforts on this, I tracked down some other issues and maybe this fixes yours as well. I'll look into it soon and report progress, hopefully we can nail that for 1.4.3.




[JCBC-466] Request full dataset when working with development views Created: 06/Jun/14  Updated: 10/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Core, Views
Affects Version/s: 1.4.1
Fix Version/s: .next
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Philipp Fehre Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Currently there is no way to request a full dataset when working with development views from the Java Client, this is solvable by just using the Production view and changing the path as well as appending the needed query parameter, but it would be nice to have the option for development views.

This came up and was requested at the Training in NY after Couchbase Live.




[JCBC-465] Unexpected behaviour on increment / decrement when combining with lock Created: 06/Jun/14  Updated: 10/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.4.1
Fix Version/s: .next
Security Level: Public

Type: Bug Priority: Trivial
Reporter: Philipp Fehre Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When issuing an increment or decrement on a locked key (via getAndLock) the result is -1, which is according to documentation that the key hasn't been found. This is confusing when combining with default values. I created a gist for this to illustrate https://gist.github.com/sideshowcoder/8d3ae572bc8e0742ae64

I think a solution would be to add this to the documentation as expected.

This came up in the training in NY.




[JCBC-464] Client never recovers when all nodes are shut down Created: 03/Jun/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File thread-dump.txt    
Issue Links:
Relates to

 Description   
When running this demo, if I disconnect the cluster entirely and restart it, the client reconnects but never recovers:
https://github.com/couchbaselabs/CBLiveDemo

 Comments   
Comment by Michael Nitschinger [ 14/Jul/14 ]
http://review.couchbase.org/#/c/39346/




[JCBC-463] Java client occasionally leaks (non-daemon) netty IO threads after shutdown Created: 27/May/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Task Priority: Major
Reporter: Dave Engberg Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We're using CouchbaseClient v1.4 from a Java server with moderately high concurrency. When we shutdown the Couchbase client, it usually comes down cleanly, but we occasionally see lingering Netty threads
{noformat}
"New I/O worker #1" prio=10 tid=0x00000000014a3000 nid=0x58bd runnable [0x00007fc7c2f88000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059a590988> (a sun.nio.ch.Util$2) - locked <0x000000059a5909a0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a5af320> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:223) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) "Memcached IO over {MemcachedConnection to /10.6.70.3:11210 /10.6.70.2:11210 /10.6.70.1:11210}" prio=10 tid=0x00007fc7dcd48800 nid=0x58b8 runnable [0x00007fc7c38f1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059e331fe0> (a sun.nio.ch.Util$2) - locked <0x000000059e331ff8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a513d90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:222) at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158)
{noformat}

This prevents clean Java server shutdown, and requires an aggressive "kill -9" to clean up. We figured out how to make the second thread start as 'daemon', but the "New I/O worker" thread can't be converted to daemon without gutting the client libs pretty heavily.

I haven't been able to reproduce this in isolated unit tests. We (Evernote) run around 500 Java7+Tomcat servers that receive a lot of activity over the course of a week, and see this problem come up in the wild around 20% of the time when we try to shut down. So it's a bit hard to narrow down the exact conditions that cause the thread leakage.

Our current workaround:
{noformat}
--- /tmp/BucketMonitor.java 2014-05-16 10:04:58.000000000 -0700
+++ src/main/java/com/couchbase/client/vbucket/BucketMonitor.java 2014-05-16 10:04:16.000000000 -0700
@@ -97,13 +97,28 @@
     this.configParser = configParser;
     this.host = cometStreamURI.getHost();
     this.port = cometStreamURI.getPort() == -1 ? 80 : cometStreamURI.getPort();
- factory = new NioClientSocketChannelFactory(Executors.newCachedThreadPool(),
- Executors.newCachedThreadPool());
+ factory = new NioClientSocketChannelFactory(newThreadPool(),
+ newThreadPool());
     this.headers = new HttpMessageHeaders();
       this.provider = provider;
   }
 
   /**
+ * Creates an executor based on a simple thread pool that only
+ * uses 'daemon' threads.
+ */
+ private java.util.concurrent.Executor newThreadPool() {
+ return Executors.newCachedThreadPool(
+ new java.util.concurrent.ThreadFactory() {
+ public Thread newThread(Runnable r) {
+ Thread t = new Thread(r);
+ t.setDaemon(true);
+ return t;
+ }
+ });
+ }
+
+ /**
    * Take any action required when the monitor appears to be disconnected.
{noformat}

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
I've got a similar report with the memcache IO thread, maybe this is related. I'll dig into it further and update the progress.
Comment by Michael Nitschinger [ 02/Jun/14 ]
I started work in both spy and jcbc to tackle this individually and harden the shutdown procedures in both.

http://review.couchbase.org/#/c/37726/
http://review.couchbase.org/#/c/37724/




[JCBC-461] Use alternative constructor to allow configuration of max New I/O client Worker thread count Created: 21/May/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Vanessa Ablaya Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Linux, Java 1.6


 Description   
We are running multiple instances of an application on a blade with 24 cores. Netty creates (2 * #PROCESSORS) New I/O Client Worker threads per couchbase client. This is causing high thread count, as each of our application instances does not have complete access to the resources 24 processors. I want to be able to configure the maximum number of threads Netty creates.

Line 100 of BucketMonitor.java uses the following constructor from the netty library:

 public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor)


I would like it to use the alternative constructor that netty provides, so that I can configure the workerCount value:

public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor, int workerCount)



 Comments   
Comment by Michael Nitschinger [ 22/May/14 ]
That makes total sense. I'd rather not add an additional config setting for 1.* since the whole thing is revamaped for the 2.0 where the complete IO layer is based on netty and fully tunable.

What I could imagine though is a system property that you can set, which we could document. So people which run on those very powerful boxes can set it, and the others just don't have to care.
What do you think?
Comment by Michael Nitschinger [ 02/Jun/14 ]
Actually, I think we can trim that down since we are using it for streaming connections only. No need to provide that much. I'll investigate further and see what I can get into 1.4.2.
Comment by Michael Nitschinger [ 02/Jun/14 ]
Btw, two more things:

1) If you go to 1.4.1 and higher, it uses a different config fetching mechanism, so you should not see any of those threads created if you go with a server 2.5.0 and higher.
2) Even on your version, netty should create those threads only when needed, so do you mean you create lots of CouchbaseClient instances? You should reuse them as a singleton. Or do you see lots of those created even for one CouchbaseClient object? If so, this is a bug I think.

Can you provide more details on that? Maybe with a thread dump?
Comment by Michael Nitschinger [ 02/Jun/14 ]
http://review.couchbase.org/#/c/37722/
Comment by Michael Nitschinger [ 02/Jun/14 ]
We know only use one, so no need to configure. Please create a new ticket if you still then see issues with massive amount of threads generated, this is something else then for sure.




[JCBC-460] New configuration gets falsely discarded if only replica vbuckets change. Created: 12/May/14  Updated: 16/May/14  Resolved: 16/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-459] Ignore Configurations with a smaller or equal revid (if set) Created: 12/May/14  Updated: 16/May/14  Resolved: 16/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0, 1.4.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-458] Improve logging Created: 08/May/14  Updated: 12/May/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Patrick Varley Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: logging
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
A few suggestion on making the log files easier to read and grep:

All messages should be pre fixed with the bucket they are related to. This is extremely important when a user has a number of different couchbase buckets connecting to different clusters but all logging to one place.

When the client starts for the 1st time it should log its own version.

What did the state change from and to?
Connection state changed for sun.nio.ch.SelectionKeyImpl@51ec8098

What is Binary config (I assume carrier publication)?
Binary config not available, bootstrapped through HTTP.




[JCBC-457] Force carrier config fetching on reconnect of node. Created: 07/May/14  Updated: 12/May/14  Resolved: 08/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-456] view query hangs indefinitely, prevents client shutdown too Created: 06/May/14  Updated: 12/May/14  Resolved: 12/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Mac OS X, Java 1.7.0_55 (I think)


 Description   
Grab the Developer Day code at https://github.com/couchbaselabs/DeveloperDay

Update the client to version 1.4.0.

Run example #9.

For me, it hangs after the last view request. If I downgrade to 1.3.2, it does not hang.

 Comments   
Comment by Matt Ingenthron [ 12/May/14 ]
verified fixed in 1.4.1




[JCBC-455] ClusterManager to have the same functionally as the UI. Created: 30/Apr/14  Updated: 12/May/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Patrick Varley Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: cluster, customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 1.4

Issue Links:
Dependency

 Description   
It would be good if the ClusterManager class could do everything that the user could do via the web interface. For example the createNamedBucket() cannot set the number of R/W threads right now.

I guess the one problem with this is what to do when features get add/removed from the cluster? (May a subclass per a cluster version?)




[JCBC-454] Query View bug Created: 27/Apr/14  Updated: 05/May/14  Resolved: 05/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.0
Fix Version/s: 1.4.1
Security Level: Public

Type: Bug Priority: Critical
Reporter: Nas Kavian Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File stack trace.png    
Issue Links:
Duplicate
duplicates SPY-163 Bulk latch never counts down on empty... Resolved

 Description   
1.3.2 works, but 1.4.0 is broken.

Perform a simple query where you set the key to something non-existent and the query will never return. I think an empty result set is properly returned from the server but the Java client doesn't handle it properly afterwards.

      final Query query = new Query();
      query.setIncludeDocs(true);
      query.setKey(ComplexKey.of("abc")); << Non-existent key
      query.setStale(Stale.FALSE);
      final ViewResponse response = client.query(view, query); << Doesn't return.

I think the attached stack trace gives some idea of where it's hung up.

 Comments   
Comment by Michael Nitschinger [ 05/May/14 ]
Hi,

thanks for reporting this - it is a known issue and is already fixed in master, will be in the next bugfix release 1.4.1 this week hopefully.




[JCBC-453] Also check for new configs with different failure modes. Created: 25/Apr/14  Updated: 08/May/14  Resolved: 08/May/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-452] Publish docs for Java May 2014 release Created: 22/Apr/14  Updated: 09/May/14  Resolved: 09/May/14

Status: Closed
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Task Priority: Critical
Reporter: Amy Kurtzman Assignee: Amy Kurtzman
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 1h
Time Spent: Not Specified
Original Estimate: 1h





[JCBC-451] Title on getting started refers to 1.3 Created: 18/Apr/14  Updated: 18/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.2
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Patrick Varley Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: gettingstarted
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: http://www.couchbase.com/communities/java/getting-started


 Description   
The title of the page is Couchbase Java Client Library 1.3. It should be 1.4.

 Comments   
Comment by Michael Nitschinger [ 18/Apr/14 ]
thanks!




[JCBC-450] 1.4.0 is not published on maven central Created: 17/Apr/14  Updated: 21/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 1.4.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Just reported by support, 1.4.0 doesn't seem to be on maven central.

 Comments   
Comment by Michael Nitschinger [ 18/Apr/14 ]
It shows up:

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.couchbase.client%22%20AND%20a%3A%22couchbase-client%22

and the artifacts can be downloaded.
Comment by Michael Nitschinger [ 18/Apr/14 ]
Dont think this is a bug, maybe the person needs to update his maven index in the IDE from time to time so it gets picked up properly by autocomplete.
Comment by Matt Ingenthron [ 21/Apr/14 ]
We determined that the issue is the lexical sort of the artifacts. In the future, we'll need to be more careful about how we publish DPs. I think the -SNAPSHOT approach understood by maven needs to be implemented. This should be fixed with a 1.4.1 shortly.




[JCBC-449] Add an overview to the javadoc and the CouchbaseClient class Created: 15/Apr/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The HTML generated autodocs should have a better intro and get the user to the CouchbaseClient class more directly. For instance, if you go here:
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.0/index.html

One would have to click around for a bit to find what they're searching for.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
Are you sure this is possible with javadoc? I googled around a bit and didn't find anything.. well we could make it work with changing the files after generation or so.
Comment by Michael Nitschinger [ 17/Jun/14 ]
I fixed that in my build script, check out: http://www.couchbase.com/autodocs/couchbase-java-client-1.4.2/index.html




[JCBC-448] TU1 Upgrade scenario | ConcurrentModificationException in memcached connection while checking timeout Created: 07/Apr/14  Updated: 10/Apr/14  Resolved: 10/Apr/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description