[JCBC-537] Setting Reader/Writer Worker value on Server causes ParseException on Client Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File output.log    
Issue Links:
Duplicate

 Description   
When setting the "Number of suggested reader/writer workers" in the Admin UI to 8, the 1.4.4 client then begins to see ParseException errors during the bootstrap phase. The client appears to continue as expected after that, but the error is concerning. The exception does not occur in the .NET or C client libraries.

To repro:

* Set the worker value to 8 in the UI
* Run the HelloWorld example from:

http://www.couchbase.com/communities/java/getting-started

Relevant code is:

// (Subset) of nodes in the cluster to establish a connection
    List<URI> hosts = Arrays.asList(
      new URI("http://127.0.0.1:8091/pools")
    );
 
    // Name of the Bucket to connect to
    String bucket = "default";
 
    // Password of the bucket (empty) string if none
    String password = "";
 
    // Connect to the Cluster
    CouchbaseClient client = new CouchbaseClient(hosts, bucket, password); // <- exception here

When running the example, the response is consistently:

2014-08-29 15:36:21.979 INFO net.spy.memcached.auth.AuthThread: Authenticated to /<ip_address>:11210
2014-08-29 15:36:22.014 WARN com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could not parse config, retrying bootstrap.
java.text.ParseException: A JSONObject text must begin with '{' at character 0 of
at com.couchbase.client.vbucket.config.ConfigurationParserJSON.parseBucket(ConfigurationParserJSON.java:148)



 Comments   
Comment by Matt Ingenthron [ 29/Aug/14 ]
If you crank up the log level, it should print out what the cluster is returning. Maybe the .NET and C clients are probably seeing the same behavior but not logging the warning?
Comment by Jeff Dillon [ 29/Aug/14 ]
In .NET I used the following log settings, and didn't see the error:

      <priority value="ALL" />
      <level value="DEBUG" />
Comment by Matt Ingenthron [ 29/Aug/14 ]
Yep, might be a bug that .NET doesn't log a warning on bad response and goes on to the next node.

The parse error comes from Jettison. https://github.com/couchbase/couchbase-java-client/blob/669c9580674082b2acccd769d9da5e1b3bb3e417/src/main/java/com/couchbase/client/vbucket/config/ConfigurationParserJSON.java#L148

It'd be really helpful if you could get the log to tell us what the cluster is returning.
Comment by Jeff Dillon [ 29/Aug/14 ]
Attaching Java log using:

logging.properties
handlers = java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=output.log
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
com.couchbase.client.vbucket.level = FINEST
com.couchbase.client.vbucket.config.level = FINEST
com.couchbase.client.level = FINEST




[JCBC-536] Interpretation and Performance Impact of Client Profiling Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Customer is using client side profiling as suggested in:

http://blog.couchbase.com/whats-new-couchbase-java-sdk-12

Do we have documentation on the Meters, Counters and Histogram output? What would the performance impact be, if any? Can this be safely left in place in a Production environment, for example?




[JCBC-535] Add documentation on different document types Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-534] Add documentation how JSON is handled Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-533] Add documentation for 1.x migration Created: 28/Aug/14  Updated: 28/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-532] Implement Common Flags Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Brett Lawson Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Implement the Common Flags formatting as defined by the specification.




[JCBC-509] Client doesn't shut down Created: 17/Aug/14  Updated: 27/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Adam Honen Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Java 87 update 11.
Windows 7 Service Pack 1 & Linux.

Attachments: Text File disconnection.log    
Issue Links:
Dependency

 Description   
Despite calling cluster.disconnect(), there are remaining non daemon (netty) threads which prevent the JVM from shutting down.

A thread dump from my machine when the program should have terminated:

Full thread dump Java HotSpot(TM) Client VM (25.11-b03 mixed mode):

"DestroyJavaVM" #22 prio=5 os_prio=0 tid=0x0016d800 nid=0x1f38 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"RxComputationThreadPool-4" #21 daemon prio=5 os_prio=0 tid=0x15255400 nid=0x17ac waiting on condition [0x18c7f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a02bd00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-3" #20 daemon prio=5 os_prio=0 tid=0x18390400 nid=0x7d0 waiting on condition [0x18d4f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a044918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-3" #12 prio=10 os_prio=2 tid=0x154b4c00 nid=0x1ba0 runnable [0x1730f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02e818> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a03b060> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02e7a0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-2" #11 prio=10 os_prio=2 tid=0x154a4800 nid=0x1524 runnable [0x1735f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02e400> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a038df8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02e388> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-2" #19 daemon prio=5 os_prio=0 tid=0x15374000 nid=0x1748 waiting on condition [0x14f9f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a0447a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-3-1" #18 daemon prio=1 os_prio=-2 tid=0x15336c00 nid=0x1ffc waiting on condition [0x15dcf000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.couchbase.client.deps.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"nioEventLoopGroup-2-1" #10 prio=10 os_prio=2 tid=0x152f4400 nid=0x1934 runnable [0x15c3f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0a02dfe8> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x0a036b90> (a java.util.Collections$UnmodifiableSet)
        - locked <0x0a02df70> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

"RxComputationThreadPool-1" #16 daemon prio=5 os_prio=0 tid=0x15284800 nid=0x180c waiting on condition [0x15adf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0a044860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00f74c00 nid=0xefc runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x00f65800 nid=0x1600 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x00f5f800 nid=0x1a78 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x00f5cc00 nid=0x1e54 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00f20c00 nid=0x1e30 in Object.wait() [0x14cff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x09de3798> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
        - locked <0x09de3798> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00f1a800 nid=0x1154 in Object.wait() [0x047bf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x09de3938> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x09de3938> (a java.lang.ref.Reference$Lock)

"VM Thread" os_prio=2 tid=0x00f17400 nid=0x1d7c runnable

"VM Periodic Task Thread" os_prio=2 tid=0x00f7e400 nid=0x1264 waiting on condition

JNI global references: 242


Code to reproduce:

    final ViewQuery view = ViewQuery
        .from("views", "myView")
        .startKey(JsonArray.from(5))
        .endKey(JsonArray.from(6))
        .reduce(false);

    logger.debug("view=" + view);

    final Observable<ViewResult> query = bucket.query(view);
    logger.debug("query=" + query);

    query
        .doOnNext(item -> logger.debug("item={}; item.success()={}", item, item.success()))
        .filter(item -> item.success())
        .doOnNext(item -> logger.debug("getting rows"))
        .flatMap(ViewResult::rows)
        .doOnNext(item -> logger.debug("view row={}", item))
        .filter(item -> choose(item))
        .take(howMany)
        .doOnNext(item -> log(item))
        .doOnError(e -> e.printStackTrace())
        .doOnTerminate(() -> {
          System.out.println("done");
          cluster.disconnect();
        })
        .subscribe();
    logger.debug("subscribed");

    Thread.sleep(1000);
    cluster.disconnect();

(This is called by the main thread after connecting to the cluster)

This will print done, but never terminate.

 Comments   
Comment by Michael Nitschinger [ 18/Aug/14 ]
thanks for reporting it, I'll look into it soon.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Found the issue, from the client the stateful environment was not properly shutdown. Should be fixed on master and is in the beta release. Let me know if it still not works, thanks!
Comment by Adam Honen [ 24/Aug/14 ]
Actually it did not work.
I tried with the beta jar, but I still see those Netty IO (non daemon) threads hanging around.

I tried also adding a call to bucket.close(), but that didn't help either.

Comment by Adam Honen [ 24/Aug/14 ]
Attached is the TRACE log that is emitted after I call the following two lines of code:

bucket.close();
cluster.disconnect();
Comment by Michael Nitschinger [ 27/Aug/14 ]
Ah, are you passing in a custom environment? Your demo code doesn't show the init path. If so, you need to shutdown() it manually. We can't do that on cluster.disconnect() because it could be shared across more cluster objects.
Comment by Michael Nitschinger [ 27/Aug/14 ]
Note that I made a change for beta2 so that 1) the owned pools are properly named (prefixed with cb) and are all daemonized, so even if the user doesn't call shutdown as he should at least the JVM exits.




[JCBC-500] CouchbaseClient doesn't timeout, hangs forever Created: 27/Jul/14  Updated: 26/Aug/14  Resolved: 26/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Milan Simonovic Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
On a single node cluster:

1. open CouchbaseClient with opTimeout = 10sec (on a different machine)
2. write 50 docs (works fine)
3. make the server drop packets for 15sec (iptables -A INPUT -p tcp --dport 200:65535 -j DROP)
4. try to write another doc -> blocks forever

I've tried different timeout values: opTimeout =2.5sec, dropping packets for 5sec, but it also never times-out.

The code to reproduce this issue is here: https://github.com/mbsimonovic/jepsen-couchbase.
A more detailed explanation in my blog post: http://milansimonovic.com/2014/07/22/smashing-couchbase/

 Comments   
Comment by Milan Simonovic [ 28/Jul/14 ]
tested with Java SDK v1.4.3
Comment by Michael Nitschinger [ 14/Aug/14 ]
Hey Milan,

I cloned your project and adjusted it to my 3 node cluster. What I got was:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -Didea.launcher.port=7540 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 13.app/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/michael/tmp/jepsen-couchbase/target/classes:/Users/michael/.m2/repository/com/couchbase/client/couchbase-client/1.4.3/couchbase-client-1.4.3.jar:/Users/michael/.m2/repository/io/netty/netty/3.5.5.Final/netty-3.5.5.Final.jar:/Users/michael/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/michael/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/michael/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/michael/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore-nio/4.3/httpcore-nio-4.3.jar:/Users/michael/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/michael/.m2/repository/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/michael/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/michael/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/michael/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Applications/IntelliJ IDEA 13.app/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.milansimonovic.jepsen.CouchTimeoutTest
2014-08-14 10:19:38,106 INFO [com.milansimonovic.jepsen.RemoteConnection] - 192.168.56.101
2014-08-14 10:19:38,636 INFO [com.milansimonovic.jepsen.CouchbaseClientFactory] - CouchbaseClientFactory{192.168.56.101, default, }
2014-08-14 10:19:38,783 INFO [com.couchbase.client.vbucket.provider.BucketConfigurationProvider] - Could bootstrap through carrier publication.
2014-08-14 10:19:38,800 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,800 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.102:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,801 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.103:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 10:19:38,806 INFO [com.couchbase.client.CouchbaseClient] - CouchbaseConnectionFactory{bucket='default', nodes=[http://192.168.56.101:8091/pools], order=RANDOM, opTimeout=10000, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=1000, obsTimeout=10000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-08-14 10:19:38,849 INFO [com.couchbase.client.CouchbaseClient] - viewmode property isn't defined. Setting viewmode to production mode
2014-08-14 10:19:38,849 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 10:19:39,869 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:39,870 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 10:19:39,919 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 1 in 43
2014-08-14 10:19:39,922 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 2 in 3
2014-08-14 10:19:39,923 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 3 in 1
2014-08-14 10:19:39,925 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 4 in 2
2014-08-14 10:19:39,927 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 5 in 2
2014-08-14 10:19:39,928 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 6 in 1
2014-08-14 10:19:39,930 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 7 in 2
2014-08-14 10:19:39,931 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 8 in 1
2014-08-14 10:19:39,933 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 9 in 2
2014-08-14 10:19:39,934 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 10 in 1
2014-08-14 10:19:39,936 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 11 in 2
2014-08-14 10:19:39,938 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 12 in 2
2014-08-14 10:19:39,939 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 13 in 1
2014-08-14 10:19:39,941 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 14 in 2
2014-08-14 10:19:39,943 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 15 in 2
2014-08-14 10:19:39,945 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 16 in 2
2014-08-14 10:19:39,946 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 17 in 1
2014-08-14 10:19:39,948 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 18 in 2
2014-08-14 10:19:39,949 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 19 in 1
2014-08-14 10:19:39,953 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 20 in 2
2014-08-14 10:19:39,955 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 21 in 2
2014-08-14 10:19:39,956 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 22 in 1
2014-08-14 10:19:39,958 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 23 in 2
2014-08-14 10:19:39,959 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 24 in 1
2014-08-14 10:19:39,961 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 25 in 2
2014-08-14 10:19:39,963 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 26 in 2
2014-08-14 10:19:39,964 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 27 in 1
2014-08-14 10:19:39,966 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 28 in 2
2014-08-14 10:19:39,968 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 29 in 2
2014-08-14 10:19:39,969 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 30 in 1
2014-08-14 10:19:39,971 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 31 in 2
2014-08-14 10:19:39,972 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 32 in 1
2014-08-14 10:19:39,974 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 33 in 2
2014-08-14 10:19:39,976 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 34 in 2
2014-08-14 10:19:39,977 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 35 in 1
2014-08-14 10:19:39,979 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 36 in 2
2014-08-14 10:19:39,981 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 37 in 2
2014-08-14 10:19:39,982 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 38 in 1
2014-08-14 10:19:39,984 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 39 in 2
2014-08-14 10:19:39,986 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 40 in 2
2014-08-14 10:19:39,988 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 41 in 1
2014-08-14 10:19:39,989 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 42 in 1
2014-08-14 10:19:39,991 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 43 in 1
2014-08-14 10:19:39,992 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 44 in 1
2014-08-14 10:19:39,993 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 45 in 1
2014-08-14 10:19:39,995 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 46 in 2
2014-08-14 10:19:39,996 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 47 in 1
2014-08-14 10:19:39,998 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 48 in 2
2014-08-14 10:19:39,999 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 49 in 1
2014-08-14 10:19:40,001 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 50 in 2
2014-08-14 10:19:40,001 INFO [com.milansimonovic.jepsen.Cluster] - creating 1 way partition
2014-08-14 10:19:41,003 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:41,003 INFO [com.milansimonovic.jepsen.Cluster] - creating partition DONE
2014-08-14 10:19:41,004 INFO [com.milansimonovic.jepsen.Cluster] - scheduled heal in 15 seconds
2014-08-14 10:19:56,005 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 10:19:57,007 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 10:19:57,008 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 10:19:57,008 INFO [com.milansimonovic.jepsen.Cluster] - closing connections

and the doc on the server looked like:

{
  "elements": [
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9,
    10,
    11,
    12,
    13,
    14,
    15,
    16,
    17,
    18,
    19,
    20,
    21,
    22,
    23,
    24,
    25,
    26,
    27,
    28,
    29,
    30,
    31,
    32,
    33,
    34,
    35,
    36,
    37,
    38,
    39,
    40,
    41,
    42,
    43,
    44,
    45,
    46,
    47,
    48,
    49,
    50
  ]
}

I wonder what I did different? Does it happen for you only in the single-node case? Or does running the test just not cover the bug?
Comment by Michael Nitschinger [ 14/Aug/14 ]
Ah now I get it, there should be much more appended than just 50.
Comment by Michael Nitschinger [ 14/Aug/14 ]
Ah I fixed your code. Let me do a PR for your github project.

You need to move the gets into the try/catch block as well since it can also throw a timeout (and will if there is a partition):

                CASValue<Object> casValue = client.gets(key);
                String updatedValue = appendElement(casValue.getValue().toString(), nextVal);

When I move it inside I get:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:61775,suspend=y,server=n -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/michael/tmp/jepsen-couchbase/target/classes:/Users/michael/.m2/repository/com/couchbase/client/couchbase-client/1.4.3/couchbase-client-1.4.3.jar:/Users/michael/.m2/repository/io/netty/netty/3.5.5.Final/netty-3.5.5.Final.jar:/Users/michael/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/michael/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/michael/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/Users/michael/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore/4.3/httpcore-4.3.jar:/Users/michael/.m2/repository/org/apache/httpcomponents/httpcore-nio/4.3/httpcore-nio-4.3.jar:/Users/michael/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/michael/.m2/repository/com/jcraft/jsch/0.1.51/jsch-0.1.51.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-api/1.5.10/slf4j-api-1.5.10.jar:/Users/michael/.m2/repository/org/slf4j/slf4j-log4j12/1.5.10/slf4j-log4j12-1.5.10.jar:/Users/michael/.m2/repository/log4j/log4j/1.2.15/log4j-1.2.15.jar:/Users/michael/.m2/repository/javax/mail/mail/1.4/mail-1.4.jar:/Users/michael/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Applications/IntelliJ IDEA 13.app/lib/idea_rt.jar" com.milansimonovic.jepsen.CouchTimeoutTest
Connected to the target VM, address: '127.0.0.1:61775', transport: 'socket'
2014-08-14 11:02:47,408 INFO [com.milansimonovic.jepsen.RemoteConnection] - 192.168.56.101
2014-08-14 11:02:47,771 INFO [com.milansimonovic.jepsen.CouchbaseClientFactory] - CouchbaseClientFactory{192.168.56.101, default, }
2014-08-14 11:02:47,908 INFO [com.couchbase.client.vbucket.provider.BucketConfigurationProvider] - Could bootstrap through carrier publication.
2014-08-14 11:02:47,920 INFO [com.couchbase.client.CouchbaseConnection] - Added {QA sa=/192.168.56.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2014-08-14 11:02:47,923 INFO [com.couchbase.client.CouchbaseClient] - CouchbaseConnectionFactory{bucket='default', nodes=[http://192.168.56.101:8091/pools], order=RANDOM, opTimeout=10000, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=1000, obsTimeout=10000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500}
2014-08-14 11:02:47,956 INFO [com.couchbase.client.CouchbaseClient] - viewmode property isn't defined. Setting viewmode to production mode
2014-08-14 11:02:47,956 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:02:48,970 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:48,970 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
counting down 943596453
counting down 99276090
2014-08-14 11:02:49,042 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 1 in 61
counting down 1076537872
2014-08-14 11:02:49,048 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 2 in 6
2014-08-14 11:02:49,054 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 3 in 6
counting down 1413217636
2014-08-14 11:02:49,060 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 4 in 6
counting down 491904845
2014-08-14 11:02:49,066 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 5 in 6
counting down 1312922328
2014-08-14 11:02:49,072 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 6 in 6
counting down 1529791274
2014-08-14 11:02:49,078 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 7 in 6
counting down 1349895670
2014-08-14 11:02:49,083 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 8 in 5
counting down 1129169666
2014-08-14 11:02:49,089 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 9 in 6
counting down 26740457
counting down 746758755
2014-08-14 11:02:49,094 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 10 in 5
counting down 1386645616
2014-08-14 11:02:49,100 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 11 in 5
2014-08-14 11:02:49,106 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 12 in 6
counting down 2145256693
2014-08-14 11:02:49,111 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 13 in 5
counting down 1051781801
2014-08-14 11:02:49,117 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 14 in 6
counting down 119068326
2014-08-14 11:02:49,123 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 15 in 6
counting down 1637559101
2014-08-14 11:02:49,128 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 16 in 5
counting down 1861050967
2014-08-14 11:02:49,134 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 17 in 6
counting down 1462839425
2014-08-14 11:02:49,140 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 18 in 6
counting down 383609127
2014-08-14 11:02:49,147 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 19 in 6
counting down 774334978
2014-08-14 11:02:49,156 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 20 in 8
counting down 735922763
2014-08-14 11:02:49,162 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 21 in 6
counting down 1970674321
2014-08-14 11:02:49,170 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 22 in 7
counting down 136298825
2014-08-14 11:02:49,177 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 23 in 7
counting down 494783794
2014-08-14 11:02:49,185 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 24 in 8
counting down 1546867652
2014-08-14 11:02:49,192 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 25 in 7
counting down 433354817
2014-08-14 11:02:49,199 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 26 in 7
counting down 1865938875
2014-08-14 11:02:49,207 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 27 in 8
counting down 483439410
2014-08-14 11:02:49,214 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 28 in 7
counting down 1513848351
2014-08-14 11:02:49,221 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 29 in 7
counting down 1612120937
2014-08-14 11:02:49,228 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 30 in 7
counting down 810400511
2014-08-14 11:02:49,235 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 31 in 7
counting down 689821415
2014-08-14 11:02:49,242 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 32 in 7
counting down 685219421
2014-08-14 11:02:49,248 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 33 in 6
counting down 1887689829
counting down 759433928
2014-08-14 11:02:49,254 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 34 in 6
2014-08-14 11:02:49,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 35 in 5
counting down 1781091177
2014-08-14 11:02:49,266 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 36 in 7
counting down 2123891975
2014-08-14 11:02:49,271 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 37 in 5
counting down 278306373
2014-08-14 11:02:49,278 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 38 in 6
counting down 495568085
2014-08-14 11:02:49,284 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 39 in 6
counting down 1029420558
2014-08-14 11:02:49,290 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 40 in 5
counting down 55913631
2014-08-14 11:02:49,296 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 41 in 6
counting down 664141318
2014-08-14 11:02:49,302 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 42 in 6
counting down 1953048909
2014-08-14 11:02:49,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 43 in 5
counting down 350362120
counting down 1048178143
2014-08-14 11:02:49,313 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 44 in 6
2014-08-14 11:02:49,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 45 in 5
counting down 907297240
2014-08-14 11:02:49,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 46 in 5
counting down 193602838
2014-08-14 11:02:49,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 47 in 6
counting down 255614005
2014-08-14 11:02:49,335 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 48 in 6
counting down 2128240408
2014-08-14 11:02:49,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 49 in 5
counting down 1783946138
2014-08-14 11:02:49,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 50 in 6
2014-08-14 11:02:49,347 INFO [com.milansimonovic.jepsen.Cluster] - creating 1 way partition
2014-08-14 11:02:50,348 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:02:50,348 INFO [com.milansimonovic.jepsen.Cluster] - creating partition DONE
2014-08-14 11:02:50,349 INFO [com.milansimonovic.jepsen.Cluster] - scheduled heal in 15 seconds
counting down 1330247343
2014-08-14 11:03:03,213 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - timed out adding 51
2014-08-14 11:03:03,214 ERROR [com.milansimonovic.jepsen.CouchTimeoutTest] - error adding 51
net.spy.memcached.OperationTimeoutException: Timeout waiting for value
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1147)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1211)
at com.milansimonovic.jepsen.CouchTimeoutTest.addElementToArray(CouchTimeoutTest.java:65)
at com.milansimonovic.jepsen.CouchTimeoutTest.appendToDocument(CouchTimeoutTest.java:102)
at com.milansimonovic.jepsen.CouchTimeoutTest.run(CouchTimeoutTest.java:49)
at com.milansimonovic.jepsen.CouchTimeoutTest.main(CouchTimeoutTest.java:39)
Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 192.168.56.101/192.168.56.101:11210
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:167)
at net.spy.memcached.MemcachedClient.gets(MemcachedClient.java:1137)
... 5 more
2014-08-14 11:03:05,349 INFO [com.milansimonovic.jepsen.Cluster] - healing partition
2014-08-14 11:03:06,351 DEBUG [com.milansimonovic.jepsen.RemoteConnection] - exit-status: 0
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - healing partition - DONE
2014-08-14 11:03:06,351 INFO [com.milansimonovic.jepsen.Cluster] - closing connections
counting down 1330247343
counting down 808247635
counting down 1562173079
2014-08-14 11:03:08,230 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 52 in 5014
counting down 1897031859
2014-08-14 11:03:08,236 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 53 in 6
2014-08-14 11:03:08,241 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 54 in 5
counting down 1029104896
2014-08-14 11:03:08,247 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 55 in 6
counting down 1789373824
2014-08-14 11:03:08,252 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 56 in 5
counting down 1374957093
2014-08-14 11:03:08,259 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 57 in 7
counting down 859935576
2014-08-14 11:03:08,265 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 58 in 5
counting down 959778517
2014-08-14 11:03:08,270 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 59 in 5
counting down 1856597982
2014-08-14 11:03:08,276 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 60 in 6
counting down 509403432
2014-08-14 11:03:08,283 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 61 in 7
counting down 565734120
2014-08-14 11:03:08,289 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 62 in 6
counting down 812288940
2014-08-14 11:03:08,295 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 63 in 6
counting down 362337760
2014-08-14 11:03:08,301 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 64 in 6
counting down 1815360474
counting down 238680990
2014-08-14 11:03:08,307 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 65 in 6
2014-08-14 11:03:08,312 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 66 in 5
counting down 1793734174
2014-08-14 11:03:08,318 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 67 in 6
counting down 965663094
2014-08-14 11:03:08,323 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 68 in 5
counting down 1961465726
2014-08-14 11:03:08,329 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 69 in 6
counting down 894265206
counting down 237825287
2014-08-14 11:03:08,334 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 70 in 5
2014-08-14 11:03:08,340 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 71 in 6
counting down 1093062877
2014-08-14 11:03:08,346 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 72 in 6
counting down 2043477472
2014-08-14 11:03:08,352 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 73 in 6
counting down 660886392
2014-08-14 11:03:08,358 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 74 in 6
counting down 1321850438
2014-08-14 11:03:08,363 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 75 in 5
counting down 966674298
2014-08-14 11:03:08,368 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 76 in 5
counting down 665030228
counting down 584709707
2014-08-14 11:03:08,373 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 77 in 5
2014-08-14 11:03:08,379 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 78 in 6
counting down 581070768
2014-08-14 11:03:08,384 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 79 in 5
counting down 1766611209
2014-08-14 11:03:08,390 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 80 in 6
counting down 1378265946
2014-08-14 11:03:08,395 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 81 in 5
counting down 1067480315
2014-08-14 11:03:08,400 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 82 in 5
counting down 240010064
2014-08-14 11:03:08,406 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 83 in 6
counting down 96589697
counting down 1650118680
2014-08-14 11:03:08,411 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 84 in 5
2014-08-14 11:03:08,417 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 85 in 6
counting down 1583441668
2014-08-14 11:03:08,422 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 86 in 5
counting down 575269723
2014-08-14 11:03:08,427 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 87 in 5
counting down 692018508
2014-08-14 11:03:08,434 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 88 in 6
counting down 500269667
2014-08-14 11:03:08,440 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 89 in 6
counting down 1214091162
2014-08-14 11:03:08,446 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 90 in 6
counting down 631482105
2014-08-14 11:03:08,452 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 91 in 6
counting down 1982159420
2014-08-14 11:03:08,458 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 92 in 6
counting down 1744182273
counting down 1131642426
2014-08-14 11:03:08,463 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 93 in 5
2014-08-14 11:03:08,469 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 94 in 6
counting down 2069859523
2014-08-14 11:03:08,475 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 95 in 6
counting down 662166830
2014-08-14 11:03:08,481 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 96 in 6
counting down 1882643413
2014-08-14 11:03:08,487 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 97 in 6
counting down 678230343
2014-08-14 11:03:08,493 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 98 in 6
counting down 793919773
2014-08-14 11:03:08,498 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 99 in 5
counting down 911566708
2014-08-14 11:03:08,505 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 100 in 7
counting down 1887173144
2014-08-14 11:03:08,510 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 101 in 5
counting down 1058177937
2014-08-14 11:03:08,516 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 102 in 6
counting down 41252807
2014-08-14 11:03:08,523 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 103 in 6
counting down 528932410
2014-08-14 11:03:08,531 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 104 in 8
counting down 805461224
2014-08-14 11:03:08,538 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 105 in 7
counting down 463949682
2014-08-14 11:03:08,546 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 106 in 8
counting down 851822734
2014-08-14 11:03:08,554 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 107 in 8
counting down 454797282
2014-08-14 11:03:08,561 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 108 in 7
counting down 962354549
2014-08-14 11:03:08,568 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 109 in 7
counting down 413223411
2014-08-14 11:03:08,574 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 110 in 6
counting down 79455185
2014-08-14 11:03:08,580 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 111 in 6
counting down 1113518099
2014-08-14 11:03:08,586 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 112 in 6
counting down 431139544
2014-08-14 11:03:08,592 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 113 in 6
counting down 198084300
2014-08-14 11:03:08,599 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 114 in 7
counting down 1821101711
2014-08-14 11:03:08,605 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 115 in 6
counting down 1499489637
2014-08-14 11:03:08,611 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 116 in 6
counting down 1435037947
2014-08-14 11:03:08,618 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 117 in 7
counting down 705688672
2014-08-14 11:03:08,624 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 118 in 6
counting down 407021592
2014-08-14 11:03:08,630 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 119 in 6
counting down 1121168696
2014-08-14 11:03:08,637 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 120 in 7
counting down 1347326641
counting down 87539756
2014-08-14 11:03:08,642 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 121 in 5
2014-08-14 11:03:08,647 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 122 in 5
counting down 137935610
2014-08-14 11:03:08,653 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 123 in 6
counting down 1474133160
2014-08-14 11:03:08,659 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 124 in 6
counting down 1931651064
2014-08-14 11:03:08,665 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 125 in 6
counting down 1211051792
2014-08-14 11:03:08,688 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 126 in 23
counting down 93172613
2014-08-14 11:03:08,693 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 127 in 5
counting down 965322656
2014-08-14 11:03:08,699 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 128 in 6
counting down 1601358350
2014-08-14 11:03:08,705 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 129 in 6
counting down 1855207775
2014-08-14 11:03:08,710 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 130 in 5
counting down 107851411
2014-08-14 11:03:08,716 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 131 in 6
counting down 1915600217
2014-08-14 11:03:08,721 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 132 in 5
counting down 1002847144
2014-08-14 11:03:08,727 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 133 in 6
counting down 1960289102
2014-08-14 11:03:08,732 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 134 in 5
counting down 530940529
2014-08-14 11:03:08,738 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 135 in 6
counting down 16593833
2014-08-14 11:03:08,744 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 136 in 6
counting down 183255909
2014-08-14 11:03:08,749 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 137 in 5
counting down 1139297445
counting down 1886255267
2014-08-14 11:03:08,755 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 138 in 5
2014-08-14 11:03:08,760 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 139 in 5
counting down 426044796
2014-08-14 11:03:08,765 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 140 in 5
counting down 1903915614
2014-08-14 11:03:08,770 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 141 in 5
counting down 1895269808
2014-08-14 11:03:08,776 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 142 in 6
counting down 1866519533
counting down 1366412404
2014-08-14 11:03:08,781 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 143 in 5
2014-08-14 11:03:08,786 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 144 in 5
counting down 2075311494
2014-08-14 11:03:08,792 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 145 in 6
counting down 925138245
2014-08-14 11:03:08,797 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 146 in 5
counting down 1578242039
2014-08-14 11:03:08,802 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 147 in 5
counting down 23034413
2014-08-14 11:03:08,808 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 148 in 6
counting down 1869588749
counting down 2067087344
2014-08-14 11:03:08,814 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 149 in 6
2014-08-14 11:03:08,820 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 150 in 6
counting down 899087062
2014-08-14 11:03:08,826 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 151 in 6
counting down 1340235867
2014-08-14 11:03:08,832 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 152 in 6
counting down 1574412139
2014-08-14 11:03:08,838 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 153 in 6
counting down 256897438
2014-08-14 11:03:08,844 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 154 in 6
counting down 227404523
2014-08-14 11:03:08,849 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 155 in 5
counting down 1370600377
2014-08-14 11:03:08,855 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 156 in 5
counting down 821838371
2014-08-14 11:03:08,861 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 157 in 6
counting down 1087346339
counting down 578287331
2014-08-14 11:03:08,866 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 158 in 5
2014-08-14 11:03:08,872 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 159 in 6
counting down 2114430319
2014-08-14 11:03:08,877 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 160 in 5
counting down 750263766
2014-08-14 11:03:08,883 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 161 in 6
counting down 285142124
2014-08-14 11:03:08,889 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 162 in 6
counting down 228835454
2014-08-14 11:03:08,895 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 163 in 6
counting down 1322000504
2014-08-14 11:03:08,900 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 164 in 5
counting down 578208474
2014-08-14 11:03:08,906 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 165 in 6
counting down 1158469492
counting down 1446993819
2014-08-14 11:03:08,911 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 166 in 5
2014-08-14 11:03:08,917 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 167 in 6
counting down 2036300484
2014-08-14 11:03:08,923 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 168 in 5
counting down 1839929127
2014-08-14 11:03:08,928 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 169 in 5
counting down 1321196177
2014-08-14 11:03:08,934 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 170 in 6
counting down 1657833306
2014-08-14 11:03:08,941 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 171 in 7
counting down 1616599198
2014-08-14 11:03:08,948 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 172 in 7
counting down 1434216861
2014-08-14 11:03:08,955 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 173 in 7
counting down 1697078001
2014-08-14 11:03:08,962 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 174 in 7
counting down 53209944
2014-08-14 11:03:08,969 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 175 in 7
counting down 957271051
2014-08-14 11:03:08,983 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 176 in 14
counting down 260602422
2014-08-14 11:03:08,991 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 177 in 8
counting down 1344752100
2014-08-14 11:03:08,996 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 178 in 5
counting down 210373011
2014-08-14 11:03:09,003 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 179 in 7
counting down 1135342237
counting down 255748372
2014-08-14 11:03:09,008 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 180 in 5
2014-08-14 11:03:09,013 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 181 in 5
counting down 1706850686
2014-08-14 11:03:09,018 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 182 in 5
counting down 958800587
2014-08-14 11:03:09,023 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 183 in 5
counting down 621084776
counting down 1494203157
2014-08-14 11:03:09,028 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 184 in 5
2014-08-14 11:03:09,034 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 185 in 6
counting down 1610501827
2014-08-14 11:03:09,039 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 186 in 5
counting down 1472601997
2014-08-14 11:03:09,046 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 187 in 6
counting down 1523401565
2014-08-14 11:03:09,052 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 188 in 6
counting down 2118032386
2014-08-14 11:03:09,058 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 189 in 6
counting down 1402690896
2014-08-14 11:03:09,063 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 190 in 5
counting down 877005529
counting down 69300391
2014-08-14 11:03:09,068 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 191 in 5
counting down 250950317
2014-08-14 11:03:09,074 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 192 in 6
2014-08-14 11:03:09,080 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 193 in 6
counting down 126717950
2014-08-14 11:03:09,086 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 194 in 6
counting down 60378388
2014-08-14 11:03:09,092 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 195 in 6
counting down 213037448
2014-08-14 11:03:09,098 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 196 in 6
counting down 737512169
2014-08-14 11:03:09,103 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 197 in 5
counting down 1089208854
2014-08-14 11:03:09,109 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 198 in 6
counting down 156501746
2014-08-14 11:03:09,115 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 199 in 6
counting down 284416161
2014-08-14 11:03:09,121 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - added 200 in 6
2014-08-14 11:03:09,123 INFO [com.couchbase.client.CouchbaseConnection] - Shut down Couchbase client
2014-08-14 11:03:09,125 INFO [com.couchbase.client.ViewConnection] - I/O reactor terminated
2014-08-14 11:03:09,125 INFO [com.milansimonovic.jepsen.CouchTimeoutTest] - DONE!
Disconnected from the target VM, address: '127.0.0.1:61775', transport: 'socket'

Process finished with exit code 0

Comment by Michael Nitschinger [ 14/Aug/14 ]
https://github.com/mbsimonovic/jepsen-couchbase/pull/2
Comment by Matt Ingenthron [ 18/Aug/14 ]
Milan: would be great if you could update your blog with a paragraph or so at the top clarifying that it did work so people don't have to read into the comments to find that it was a problem with the test.
Comment by Milan Simonovic [ 19/Aug/14 ]
Sure, as soon as I get back from holidays (next we latest). Ps the tracker is major pain in the a** to use on a smartphone..
Comment by Milan Simonovic [ 26/Aug/14 ]
updated, feel free to close the issue
Comment by Michael Nitschinger [ 26/Aug/14 ]
Not an issue.




[JCBC-531] Expose Diagnostics and debug dump them on startup Created: 26/Aug/14  Updated: 26/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-521] VBucketNodeLocator.java getPrimary() and getReplica() log messages are homologous Created: 21/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
VBucketNodeLocator.java has these two methods:
- getPrimary(String k)
- getReplica(String key, int index)

These two write almost the same warning log message when it can't access a node which holds the key.

getPrimary() logs: "for which no server is responsible in the cluster map (-1)."
while
getReplica() logs: "for which no server is responsible in the cluster map."
(without -1 in the message)

 Comments   
Comment by Koji Kawamura [ 21/Aug/14 ]
These two methods log almost the same warning messages, but since those two are indicating different level of severity, those better to be differentiated.
Comment by Michael Nitschinger [ 25/Aug/14 ]
I think it would be okay to push the replica one down to INFO level.Would that be sufficient? I'd rather not change the message itself in a bugfix release.
Comment by Koji Kawamura [ 25/Aug/14 ]
Changing log level of missing replica node to INFO sounds reasonable. Thank you!




[JCBC-530] Exception when using java client both in Spark and MapReduce Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sun Chen Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: Hadoop,Dependencies,HttpCore, Spark,
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h
Environment: Haddop2.4.1,Spark0.13,Java1.7,Ubuntu12.04


 Description   
I am using Couchbase java client(version 1.4.4),I writed some simple mapreaduce job and spark job to write and read data from Couchbase.But I always got an exception.Here is the exception information

-------------------------------------------------------------------------------------------------------------------------------------------------
2014-08-25 13:12:56,048 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V
at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156)
at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:31)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:20)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
-------------------------------------------------------------------------------------------------------------------------------------------

14/08/09 10:19:32 WARN Utils: Your hostname, precise64 resolves to a loopback address: 127.0.1.1; using 192.168.33.10 instead (on interface eth1) 14/08/09 10:19:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/08/09 10:19:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-09 10:19:40.569 INFO net.spy.memcached.auth.AuthThread: Authenticated to /127.0.0.1:11210 2014-08-09 10:19:40.677 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication. 2014-08-09 10:19:40.693 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=localhost/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue 2014-08-09 10:19:40.702 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{bucket='Result', nodes=[http://127.0.0.1:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500} Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156) at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194) at com.eyespage.dc.etl.c2s.Run$.main(main.scala:16) at com.eyespage.dc.etl.c2s.Run.main(main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2014-08-09 10:19:40.836 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210 - See more at: http://www.couchbase.com/communities/q-and-a/exception-when-using-scala-call-java-sdk%EF%BC%8Cand-run-spark#sthash.CmL5dcRz.dpuf








[JCBC-502] Sometimes couchbase client stuck Created: 29/Jul/14  Updated: 24/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Mac OSX as client


 Description   
Sometimes I run the client like in the examples in the community blog and the client stuck and Doesn't return. To validate that I run the client several times one after the the other and sometimes it get stuck and some don't

 Comments   
Comment by Michael Nitschinger [ 11/Aug/14 ]
Can you provide logging with level set to FINEST? that would be of great help, thanks.
Comment by Oded Hassidi [ 12/Aug/14 ]
I am running my test in a main, and I have debug level entries. Is that enough?
Comment by Michael Nitschinger [ 12/Aug/14 ]
Yes send over whatever you have, and the code would also be good.
Comment by Oded Hassidi [ 12/Aug/14 ]
Code:


public class Main {
public static void main(String[] args) throws IOException {
      
        System.setProperty("com.couchbase.client.queryEnabled", "true");

        CouchbaseCluster cc = new CouchbaseCluster("obbe-centos3.tlv.lpnet.com", "obbe-centos1.tlv.lpnet.com", "obbe-centos2.tlv.lpnet.com");

        final Bucket[] bucket = new CouchbaseBucket[1];
        System.out.println("******** Opening the Bucket");
        cc.openBucket("visitors-odedh").subscribe(b -> {
            System.out.println("################### Bucket is opened: ");
                System.out.println(b.toString());
                bucket[0] = b;
        });


        System.out.println("enter something: ");
        System.in.read();
    }
}



log:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -Didea.launcher.port=7534 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 14 EAP.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dal/target/classes:/Users/odedh/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar:/Users/odedh/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar:/Users/odedh/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dao/target/classes:/Users/odedh/.m2/repository/com/liveperson/framework/rest-api-data-objects/2.0.2.11/rest-api-data-objects-2.0.2.11.jar:/Users/odedh/.m2/repository/com/liveperson/RestApiCommon/1.0.2-6/RestApiCommon-1.0.2-6.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-core/1.9.1/jersey-core-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-client/1.9.1/jersey-client-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-server/1.9.1/jersey-server-1.9.1.jar:/Users/odedh/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-json/1.9.1/jersey-json-1.9.1.jar:/Users/odedh/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/odedh/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/odedh/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/odedh/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/odedh/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/odedh/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.3/jackson-core-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.3/jackson-mapper-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/Users/odedh/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar:/Users/odedh/.m2/repository/org/springframework/spring-context/4.0.6.RELEASE/spring-context-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-aop/4.0.6.RELEASE/spring-aop-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/odedh/.m2/repository/org/springframework/spring-beans/4.0.6.RELEASE/spring-beans-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-core/4.0.6.RELEASE/spring-core-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-expression/4.0.6.RELEASE/spring-expression-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/couchbase/client/couchbase-client/2.0.0-dp2/couchbase-client-2.0.0-dp2.jar:/Users/odedh/.m2/repository/com/couchbase/client/core-io/0.2/core-io-0.2.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.4.0/jackson-databind-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.0/jackson-annotations-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.0/jackson-core-2.4.0.jar:/Users/odedh/.m2/repository/com/lmax/disruptor/3.2.1/disruptor-3.2.1.jar:/Users/odedh/.m2/repository/com/netflix/rxjava/rxjava-core/0.19.2/rxjava-core-0.19.2.jar:/Users/odedh/.m2/repository/com/typesafe/config/1.2.1/config-1.2.1.jar:/Users/odedh/.m2/repository/io/netty/netty-all/4.0.21.Final/netty-all-4.0.21.Final.jar:/Users/odedh/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/odedh/.m2/repository/org/springframework/spring-test/4.0.6.RELEASE/spring-test-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring/4.0.0.1/SnmpMonitoring-4.0.0.1.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring-api/4.0.0.1/SnmpMonitoring-api-4.0.0.1.jar:/Users/odedh/.m2/repository/org/apache/xmlbeans/xmlbeans/2.4.0/xmlbeans-2.4.0.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus/3.0.0.2/ServiceStatus-3.0.0.2.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus-API/3.0.0.2/ServiceStatus-API-3.0.0.2.jar:/Applications/IntelliJ IDEA 14 EAP.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.liveperson.visitorfeed.dal.couchbase.CouchbaseVisitorDao
15:55:05.598 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
15:55:05.604 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 733553586
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/5f/wk7ckrbx1f30521n8rtxndzwnvk8xk/T (java.io.tmpdir)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
15:55:05.712 [main] DEBUG c.c.c.c.config.ConfigurationProvider - Setting seed hosts to [obbe-centos1.tlv.lpnet.com/192.168.15.120, obbe-centos3.tlv.lpnet.com/192.168.15.190, obbe-centos2.tlv.lpnet.com/192.168.15.2]
******** Opening the Bucket
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
enter something:
15:55:05.780 [RxComputationThreadPool-1] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x9729d41313197dd7
15:55:05.785 [RxComputationThreadPool-1] DEBUG i.n.channel.ChannelOutboundBuffer - -Dio.netty.threadLocalDirectBufferSize: 65536
15:55:05.786 [RxComputationThreadPool-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
15:55:05.795 [RxComputationThreadPool-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
15:55:05.880 [nioEventLoopGroup-2-2] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple
15:55:05.981 [nioEventLoopGroup-2-2] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-6] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-3] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.984 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.984 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.984 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos1.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos3.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.985 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos2.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-1] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.994 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:05.996 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:06.238 [RxComputationThreadPool-2] DEBUG c.c.c.c.config.ConfigurationProvider - Applying new configuration DefaultCouchbaseBucketConfig{name='visitors-odedh', locator=null, uri='/pools/default/buckets/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', streamingUri='/pools/default/bucketsStreaming/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', nodeInfo=[NodeInfo{viewUri='http://obbe-centos1.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos1.tlv.lpnet.com/192.168.15.120, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos2.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos2.tlv.lpnet.com/192.168.15.2, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos3.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos3.tlv.lpnet.com/192.168.15.190, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}], partitionInfo=PartitionInfo{numberOfReplicas=1, partitionHosts=[obbe-centos1.tlv.lpnet.com, obbe-centos2.tlv.lpnet.com, obbe-centos3.tlv.lpnet.com], partitions=[[m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]]], forwardPartitions=[]}, tainted=false, rev=2070}
15:55:06.256 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8091
15:55:06.257 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos2.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8091
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos1.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8092
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos2.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8092
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos3.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8092
15:55:06.259 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8093
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8093
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos2.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8093
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8091
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
Comment by Michael Nitschinger [ 21/Aug/14 ]
Hi,

can you please retry with dp3 and see if the issue still persists? If so, please also provide the debug logs - thanks! Also, can you specify what "hangs" means? Your println does never show?
Comment by Oded Hassidi [ 24/Aug/14 ]
Thanks, yes my println doesn't ever show!
Will check it with the beta, sorry for the delay I was OOO couple of days :)




[JCBC-485] JsonObject and all its similars Created: 06/Jul/14  Updated: 24/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: json
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Why should I work with JsonObject, JsonArray and all of that? Why can't I just send a String and you guys do with it what ever you want? Or at least give us a utility to convert our Pojo to your JsonObject?

If there is such utility I take it back :)

 Comments   
Comment by Michael Nitschinger [ 07/Jul/14 ]
Hi,

yes you are right, and we will provide that functionality as well once we get to feature completeness on 2.0. There has already been a user contribution to a converter which will act like the one on 1.4 so you can then pass in a string and it will be stored. So you get the both of two worlds.

Does that work for you?
Comment by Michael Nitschinger [ 21/Aug/14 ]
We have implemented a LegacyDocument where you can pass in a string and it works like the old one. We probably add more documents (string,...) directly so you can have more fun with it.

I'll close this one for now since it works with beta/dp3 already using the legacy codec and open more issues for specific transcoders/documents.
Comment by Oded Hassidi [ 24/Aug/14 ]
Thanks




[JCBC-529] Add support for spatial Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-526] Add support for bucket management operations Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-527] Add cluster info Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-528] Add shortcut JsonObject and JsonArray factories Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-524] Add support for custom transcoders Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-525] Add more transcoders for basic types Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-523] Add support for memcached buckets Created: 22/Aug/14  Updated: 22/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-513] Implement bucket.close() Created: 18/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-522] Implement bucket info. Created: 22/Aug/14  Updated: 22/Aug/14  Resolved: 22/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-479] ComparisonFailure in running unit tests for 2.0 client Created: 24/Jun/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 2.0
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test

com.couchbase.client.java.document.json.JsonObjectTest > shouldExportMix FAILED
    org.junit.ComparisonFailure at JsonObjectTest.java:27

com.couchbase.client.java.query.ViewQueryTest > shouldHandleEndKey FAILED
    org.junit.ComparisonFailure at ViewQueryTest.java:178

77 tests completed, 2 failed
:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///root/couchbase/sdkd-2.0/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED


 Comments   
Comment by Wei-Li Liu [ 15/Jul/14 ]
I am getting different failure running unit test

===================================================================================
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test

com.couchbase.client.java.convert.JacksonJsonConverterTest > shouldEncodeSimpleJsonObject FAILED
    org.junit.ComparisonFailure at JacksonJsonConverterTest.java:70

com.couchbase.client.java.convert.JacksonJsonConverterTest > shouldEncodeMixedNestedJsonValues FAILED
    org.junit.ComparisonFailure at JacksonJsonConverterTest.java:177

77 tests completed, 2 failed
:test FAILED

FAILURE: Build failed with an exception.
Comment by Wei-Li Liu [ 15/Jul/14 ]
Failure running integration tests


============================================================================
wei-lis-mbp:couchbase-java-client wei-li$ ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava UP-TO-DATE
:processIntegrationResources UP-TO-DATE
:integrationClasses UP-TO-DATE
:integrationTest

com.couchbase.client.java.BinaryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.BinaryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.QueryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.QueryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.ViewTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.ViewTest > classMethod FAILED
    java.util.NoSuchElementException

6 tests completed, 6 failed
:integrationTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integrationTest'.
> There were failing tests. See the report at: file:///Users/wei-li/repo/couchbase/sdkd-2.0/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
Comment by Wei-Li Liu [ 15/Jul/14 ]
Disregard the above integration test failure. It requires to create "default" bucket and set no password to run.
Comment by Wei-Li Liu [ 18/Aug/14 ]
Running 2.0-dp3 feature test against 1118 server, and all of the unit test and integration tests passed. I am resolving this ticket.
Comment by Deepti Dawar [ 19/Aug/14 ]
Hi Michael, Please close this when you push the fix for it.

Thanks !
Comment by Michael Nitschinger [ 19/Aug/14 ]
Should be fixed in master, please reopen if not.
Comment by Wei-Li Liu [ 20/Aug/14 ]
I run int test on master today and there is 1 failure on shouldUpsertLegacyObjectDocument

com.couchbase.client.java.BinaryTest STANDARD_ERROR
    log4j:WARN No appenders could be found for logger (com.couchbase.client.core.logging.CouchbaseLoggerFactory).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument FAILED
    java.lang.NullPointerException
        at com.couchbase.client.java.BinaryTest.shouldUpsertLegacyObjectDocument(BinaryTest.java:295)
Gradle Worker 1 finished executing tests.

28 tests completed, 1 failed
Comment by Michael Nitschinger [ 21/Aug/14 ]
Hm there might be an issue with the new legacy document, a different test is failing for me.

Is there any other info in the logs what's going on?
Comment by Deepti Dawar [ 21/Aug/14 ]
The unit tests now run fine.
Tried with the latest of DP3.
Comment by Wei-Li Liu [ 21/Aug/14 ]
stacktrace for legacy document test

11:40:54.879 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument STARTED
11:40:54.918 [DEBUG] [TestEventLogger]
11:40:54.918 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldUpsertLegacyObjectDocument FAILED
11:40:54.918 [DEBUG] [TestEventLogger] java.lang.NullPointerException
11:40:54.918 [DEBUG] [TestEventLogger] at com.couchbase.client.java.BinaryTest.shouldUpsertLegacyObjectDocument(BinaryTest.java:295)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.919 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.919 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
11:40:54.919 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
11:40:54.920 [DEBUG] [TestEventLogger] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
11:40:54.921 [DEBUG] [TestEventLogger] at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
11:40:54.921 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.921 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.922 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
11:40:54.922 [DEBUG] [TestEventLogger] at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
11:40:54.922 [DEBUG] [TestEventLogger] at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11:40:54.922 [DEBUG] [TestEventLogger] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11:40:54.923 [DEBUG] [TestEventLogger] at java.lang.reflect.Method.invoke(Method.java:483)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
11:40:54.923 [DEBUG] [TestEventLogger] at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
11:40:54.923 [DEBUG] [TestEventLogger] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
11:40:54.923 [DEBUG] [TestEventLogger] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
11:40:54.923 [DEBUG] [TestEventLogger] at java.lang.Thread.run(Thread.java:745)
11:40:54.925 [DEBUG] [TestEventLogger]
11:40:54.925 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldInsertAndGet STARTED
11:40:54.925 [DEBUG] [TestEventLogger]
11:40:54.925 [DEBUG] [TestEventLogger] com.couchbase.client.java.BinaryTest > shouldInsertAndGet PASSED
11:40:57.110 [QUIET] [system.out] 11:40:57.110 [
11:40:57.110 [DEBUG] [TestEventLogger]
11:40:57.110 [QUIET] [system.out] DEBUG] [org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor] Executing test class com.couchbase.client.java.DesignDocumentTest
Comment by Wei-Li Liu [ 21/Aug/14 ]
This exception is caused by integration test, which is somewhat a duplicate of JCBC-481.
unit test works fine for me too.




[JCBC-481] Integration Tests hang up after 85% progress Created: 25/Jun/14  Updated: 21/Aug/14

Status: Reopened
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Deepti Dawar Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
[root@pomelo-11017 couchbase-java-client]# ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava UP-TO-DATE
:processIntegrationResources UP-TO-DATE
:integrationClasses UP-TO-DATE
> Building 85% > :integrationTest

 Comments   
Comment by Wei-Li Liu [ 18/Aug/14 ]
The process hangs at 85% due to view query test.
Just runs 2.0.0-dp3 integration test against the new beta2 build, and all the tests completes.
Comment by Deepti Dawar [ 21/Aug/14 ]
Following error appears with latest DP3 -

[root@pomelo-11017 couchbase-java-client]# ./gradlew integrationTest
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileIntegrationJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processIntegrationResources
:integrationClasses
:integrationTest

com.couchbase.client.java.BinaryTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.BinaryTest > classMethod FAILED
    java.util.NoSuchElementException

com.couchbase.client.java.DesignDocumentTest > classMethod FAILED
    com.couchbase.client.core.config.ConfigurationException
        Caused by: java.lang.IllegalStateException
            Caused by: rx.exceptions.OnErrorThrowable$OnNextValue

com.couchbase.client.java.DesignDocumentTest > classMethod FAILED
    java.util.NoSuchElementException

4 tests completed, 4 failed
:integrationTest FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integrationTest'.
> There were failing tests. See the report at: file:///root/couchbase/couchbase-java-client/build/reports/tests/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 22.09 secs
Comment by Michael Nitschinger [ 21/Aug/14 ]
Does it also happen on master?
Comment by Wei-Li Liu [ 21/Aug/14 ]
I test it on master and only see JCBC-479




[JCBC-499] Test SSL Feature with Java 2.0 SDK Created: 28/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Closed
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ssl.pcapng    

 Description   
Michael, here is the SSL Feature Test I created
https://github.com/couchbase/couchbase-java-client/pull/6/files
I know there are suff should be in core, and we can move it later

There are a couple issues in this feature tests

1. view and query test hangs
2. the ssl test I created is not able to catch exceptions such as sun.security.validator.ValidatorException, javax.net.ssl.SSLHandshakeException. etc

We can’t disable SSL from the server side after starting the server. SSL certificate automatically generated and self signed, so we can’t modify expired date.

I also test 2.5 server connect with java client, which does not work with SSL.

Attached file is wireshark packet capture fro FeatureTest with SSL. If you decode it with SSL in Wireshark, you can see the data and TLS protocol.
  



 Comments   
Comment by Wei-Li Liu [ 30/Jul/14 ]
java sdk's commit on observe relies on core
https://github.com/weilliu/couchbase-java-client/commit/6f8d672ff9a600780fd5d7d47d5ba6ce52bd083d

import com.couchbase.client.core.message.binary.ObserveRequest;
import com.couchbase.client.core.message.binary.ObserveResponse;

Currently Core can't be built on master

====================================
[root@centos-64-x64 couchbase-jvm-core]# ./gradlew install
:compileJava
Note: /root/sdkd-2.0/couchbase-jvm-core/src/main/java/com/couchbase/client/core/CouchbaseCore.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
:processResources
:classes
:jar
:javadoc
java.lang.UnsupportedClassVersionError: ch/raffael/doclets/pegdown/PegdownDoclet : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at com.sun.tools.javadoc.DocletInvoker.<init>(DocletInvoker.java:96)
at com.sun.tools.javadoc.Start.setDocletInvoker(Start.java:414)
at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:213)
at com.sun.tools.javadoc.Start.begin(Start.java:162)
at com.sun.tools.javadoc.Main.execute(Main.java:59)
at com.sun.tools.javadoc.Main.main(Main.java:49)
javadoc: error - fatal error
1 error
:javadoc FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':javadoc'.
> Javadoc generation failed.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 24.87 secs
====================================================

Comment by Wei-Li Liu [ 04/Aug/14 ]
Please disregard my above error, which is triggered by inconsistent jdk version on run time and compile time.
Comment by Don Pinto [ 11/Aug/14 ]
Wei-li, can you update this ticket?
Comment by Wei-Li Liu [ 11/Aug/14 ]
@Michael looking at your most recent commit, I don't think the catching exception issue has been fixed yet. I believe you just changed the fix version to 2.0-beta.
I also have some issues build couchbase-java-client from master after I pull both sdk and the core. I knew Deepti has the same issue earlier, but not sure if it has been resolved yet.
Comment by Matt Ingenthron [ 18/Aug/14 ]
Michael: this is blocking completion of testing, can you aim to resolve it this week.
Comment by Wei-Li Liu [ 18/Aug/14 ]
After getting some help from Deepti, I am able to get dp3 build with ./gradlew publish and ./gradlew publishToMavenLocal
Integration test has been changed in dp3, so some modifications required for SSL feature tests on my end.
Comment by Wei-Li Liu [ 20/Aug/14 ]
Report bug JCBC-520
Comment by Michael Nitschinger [ 21/Aug/14 ]
So are we tracking it in JCBC-520 and closing this?
Comment by Wei-Li Liu [ 21/Aug/14 ]
okay




[JCBC-514] Only shutdown environment if not shared Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-498] I can't connect to an authenticated bucket Created: 28/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Task Priority: Major
Reporter: Gareth Powell Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: MacOS X

Attachments: Text File trace.txt    

 Description   
I can't connect to a password-protected bucket with SDK 2-dp2.

"Similar" code can connect with SDK 1.4.

Here is my approximate code:

cluster = new CouchbaseCluster(couchConfig.fromHost);

Bucket bucket = cluster.openBucket(couchConfig.bucket, couchConfig.passwd).toBlocking().single();

if (couchConfig.clearDB)

bucket.flush().toBlocking().single();

where "fromHost" is set to "warburton", "bucket" is ziggrid-baseball and password is "". (Yes, that is the password on the bucket in development).

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
thanks, I'll look into it soon. Let's see if I can fix that for dp3
Comment by Gareth Powell [ 20/Aug/14 ]
I just tried this and in dp3 I can "kind of" get it to work.

The thing is, I couldn't use the code above (presumably due to API changes) and that suggests the code above might always have been invalid.

I changed:

cluster = new CouchbaseCluster(couchConfig.fromHost);

to

cluster = CouchbaseCluster.create(couchConfig.fromHost);

and

bucket.flush().toBlocking().single();

to

bucket.bucketManager().toBlocking().single().flush().toBlocking().single();
Comment by Michael Nitschinger [ 20/Aug/14 ]
Yup, your changes look good. What do you mean by kind of? Does it work with the changes?
Comment by Gareth Powell [ 20/Aug/14 ]
Yes, it works now. At the time I made the comment, I was able to connect and clear out the bucket, but nothing else appeared to be working.

The problem I'm having is the usual migration one: I have so many errors in my code from the API changing that it's hard to tell what is going wrong. It turns out that my line of execution was being sabotaged by a broken dependency that wasn't getting caught. When I was able to fix that, I was able to get to the stage of loading a design document.

I now have design documents and regular documents loading.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Perfect, thanks. Let me know if you run into more issues.




[JCBC-516] All internal code should use the provided scheduler from the environment Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Michael Nitschinger [ 21/Aug/14 ]
fixed in the core.




[JCBC-497] reduce in ViewQuery doesn't work Created: 27/Jul/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 2.0-dp2
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When querying using views, setting the reduce(false) seems to have no influence on the query and always reduces the query.

Am I missing something?

 Comments   
Comment by Michael Nitschinger [ 28/Jul/14 ]
Thanks, I'll look into it.
Comment by Michael Nitschinger [ 21/Aug/14 ]
Works in dp3 and on master!

Make sure that if you have a view with a reduce function, set .reduce(false) on the DSL to disable it optionally.




[JCBC-512] $HOST in Refresher against localhost Created: 18/Aug/14  Updated: 21/Aug/14  Resolved: 21/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
carrier (and maybe also http refresher) don't replace $HOST

 Comments   
Comment by Michael Nitschinger [ 21/Aug/14 ]
Done in core.




[JCBC-520] Running SSL Test on 2.0 Client hangs when certificate is not valid Created: 20/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The SSL test I make can be seen from
https://github.com/weilliu/couchbase-java-client/commit/97c3144692627d6e657a0cc157f5f9aa644de0f2
http://review.couchbase.org/#/c/40763/

Currently if we enable ssl, and when it hits a certificate which is not valid/not exist, the test would hang


11:38:16.868 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STARTED
11:38:17.843 [DEBUG] [TestEventLogger]
11:38:17.843 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STANDARD_OUT
11:38:17.843 [DEBUG] [TestEventLogger] File copied!
11:38:17.844 [DEBUG] [TestEventLogger] Reconnect with the Server
> 11:39:06.2 [ DEB] [org.gradle.process.internal.ExecHandleRunner] Abort requested. Destroying process: Gradle Worker 1.
> Building 85% > :integrationTest > 29 tests completed, 29 skippedWei-Lis-MacBook-Pro:couchbase-java-client wei-li$






[JCBC-519] Timeout Not Honored on First Call to Java asyncGetBulk Created: 11/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File App.java    
Issue Links:
Duplicate

 Description   
This was observed on Couchbase Server 2.2.0. The same behavior is not evident on Couchbase Server 2.5.1. It appears to be independent of any Java client version. The very first call of asyncGetBulk does not respect the timeout value, and continues to process and wait. Subsequent calls do properly respect the timeout value. One interesting note is that this only occurs when the timeout value is "close" to the actual execution time. That is, a very short timeout is indeed honored.

Steps to reproduce:

* Download a Java client, such as http://packages.couchbase.com/clients/java/1.4.3/Couchbase-Java-Client-1.4.3.zip

* Unzip

* Copy the attached App.java into the unzipped directory

* Compile and run:

javac -cp couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar App.java

java -cp .:couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar:jettison-1.1.jar:netty-3.5.5.Final.jar:commons-codec-1.5.jar:httpcore-4.3.jar:httpcore-nio-4.3.jar App > 143.txt

* Note the following output in the first group. It would be expected that the first call would timeout, but does not:

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=108ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=16ms


Exception:********************
Timeout future is: true
Operation timed out. - failing nodes: 10.4.2.107/10.4.2.107:11210, cbhost.domain/10.4.2.57:11210
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms


ASK - Determine why the first call does not honor the timeout, and ideally suggest a workaround for Couchbase Server 2.2.0, short of upgrading to Couchbase Server 2.5.1 (where the issue does not appear exist, and therefore looks to be a server-side issue)




 Comments   
Comment by Aleksey Kondratenko [ 11/Aug/14 ]
Not honoring timeouts cannot be server side issue. Passing to Michael who owns couchbase java sdk
Comment by Jeff Dillon [ 12/Aug/14 ]
Please disregard my previous comments with respect to being apparently fixed with Couchbase Server 2.5.1. This behavior evidences itself across all CB server versions, and also reproduces with CB Java client 1.4.4. In my latest 2.5.1 test, I set the timeout to 30 ms, and see the same incorrect behavior. So this is indeed a client issue (as noted), apologies on the initial misinformation.
Comment by Michael Nitschinger [ 14/Aug/14 ]
A few remarks:

- The simple benchmark also profiles logging and so forth.
- System.currentTimeMillis() precision is sometimes beyond all joke, always use nanotime.

So something like this is more accurate:

        while (low == 10) {
            try {
                startTime = System.nanoTime();
                future = client.asyncGetBulk(Main.getKeys());
                future.get(timeout, TimeUnit.MILLISECONDS);
                endTime = System.nanoTime();
                System.out.println("Timeout future is: " + future.isTimeout());
                System.out.println("Fetched " + Main.getKeys().size() + " keys");
            } catch (Exception e) {
                System.out.println("\nException:********************\nTimeout future is: " + future.isTimeout());
                System.out.println(e.getMessage());
            }
            operationTime = endTime - startTime;
            System.out.println("Timeout set to:" + timeout + "\ngetMulti : Time taken=" + TimeUnit.NANOSECONDS.toMillis(operationTime) + "ms\n");
        }

Now also note that profiling on the blocking .get() call not only waits on the latch where the timeout is applied, but it also puts all data together and so forth. So as far as I can see it the timeout is honored, its just not what you benchmark that close. Keep in mind that the first few calls are subject t JVM warmup, optimizations and so forth which can take some time. It's not a huge surprise that the first few calls are slightly slower, but I guess thats not what the JVM optimizes for compared to a long-running server.

Does that help?
Comment by Jeff Dillon [ 14/Aug/14 ]
I am seeing the same behavior when using nanoTime. And it's not slightly slower, it's an order of magnitude difference. And I don't think it's JVM warmup either, as different timeout values show consistent differences every time. Using a short time out value works as expected, but is not the timeout value they desire. When the timeout is "close" to the actual execution time, it experiences this behavior. This is what I'm seeing in my tests, and is what the customer is seeing also.
Comment by Matt Ingenthron [ 20/Aug/14 ]
Jeff Dillon: I understand from James this has been observed on 2.5.1 now as well. Can you confirm and update it appropriately? Also, I've moved it to JCBC, since the investigation has lead here.
Comment by Jeff Dillon [ 20/Aug/14 ]
Yes, please see my previous comment from 12/Aug/14 1:47 PM




[JCBC-518] no intro page on the javadoc Created: 18/Aug/14  Updated: 18/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
So it's clear where to begin working with the client.




[JCBC-517] Add overloads for Persist/Replicate only Created: 18/Aug/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-515] Simplify environment and collapse properties Created: 18/Aug/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-494] Support touch and unlock Created: 16/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-492] Support get + lock and get + touch Created: 15/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-491] Support counter Created: 15/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-489] Support Replica Read Created: 10/Jul/14  Updated: 18/Aug/14  Resolved: 18/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp3
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-510] Allow optional non-persistent view connections Created: 18/Aug/14  Updated: 18/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-246] Docs: No CAS+durability in API reference table Created: 12/Feb/13  Updated: 14/Aug/14  Resolved: 14/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.1.2
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Major
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The table here: http://www.couchbase.com/docs/couchbase-sdk-java-1.1/api-reference-summary.html

Doesn't contain CAS methods with durability constraints.

 Comments   
Comment by Matt Ingenthron [ 14/Aug/14 ]
Starting with the 1.2 client, we removed this section and went to straight javadoc. Should be resolved with that.




[JCBC-478] Docs: Deferred view deployment instruction does not work Created: 17/Feb/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.0
Fix Version/s: 2.0
Security Level: Public

Type: Bug Priority: Minor
Reporter: Don Stacy Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Section: http://docs.couchbase.com/couchbase-sdk-java-1.3/#preview-the-application
Area: Text under step 2
Issues: We say that we will deploy the views to production later. However, the app will not work unless the views are in production. We should include the steps here or at least tell the reader to publish while pointing them to the View documentation.




[JCBC-504] Couchbase hangs during tomcat shutdown Created: 30/Jul/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2, 1.4.3
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Major
Reporter: Benoit Beaudet Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Tomcat 7.0.45.


 Description   
Using couchbase lib inside a WAR. All clients connections are stopped by the application but Tomcat can't stop.
Tomcat stop process is blocked by a CouchbaseConfigConnection thread (deamon=false)

{code}
  java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    - locked <0x0000000795b80af0> (a sun.nio.ch.Util$2)
    - locked <0x0000000795b80b00> (a java.util.Collections$UnmodifiableSet)
    - locked <0x0000000795b80aa8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:420)
    at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:321)
{code}


Looking a the bootstrap code from com.couchbase.client.vbucket.provider.BucketConfigurationProvider class.
A CouchbaseConfigConnection is created but not closed if everything is working fine.
{code}
private boolean tryBinaryBootstrapForNode(InetSocketAddress node)
    throws Exception

  connection = new CouchbaseConfigConnection(
        cf.getReadBufSize(), fact, Collections.singletonList(node),
        initialObservers, cf.getFailureMode(),
        cf.getOperationFactory()
      );

{code}



Hack to resolve the issue.
{code}
  private void killConfigConnection(){
        ThreadGroup rootGroup = Thread.currentThread( ).getThreadGroup( );
        ThreadGroup parentGroup;
        while ( ( parentGroup = rootGroup.getParent() ) != null ) {
            rootGroup = parentGroup;
        }

        Thread[] threads = new Thread[ rootGroup.activeCount() ];
        while ( rootGroup.enumerate( threads, true ) == threads.length ) {
            threads = new Thread[ threads.length * 2 ];
        }

        for (int i = 0; i < threads.length; i++){
            final Thread thread = threads[i];
            if(thread instanceof CouchbaseConfigConnection){
                final CouchbaseConfigConnection configConn = (CouchbaseConfigConnection) thread;
                try {
                    configConn.shutdown();
                } catch (final IOException e) {
                   //
                }
            }

        }
   
{code}

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
Is that a duplicate of 503?
Comment by Benoit Beaudet [ 04/Aug/14 ]
This issue isn't a duplicate. A non deamon thread is still active in Tomcat. The CouchbaseConfigConnection used at boostrap should be shutdown.
Comment by Michael Nitschinger [ 05/Aug/14 ]
Hmm maybe I don't understand it. Do you call client.shutdown() or just shutdown tomcat? Because they are by intention non-daemon threads and you need to shutdown the client manually. or do you shut it down but some resources are not freed?
Comment by Benoit Beaudet [ 05/Aug/14 ]
I call client.shutdown in web context destroy event, but the CouchbaseConfigConnection isn't under my control and not freed.




[JCBC-507] Provide compatibility with 3.0 spatial views Created: 11/Aug/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In 3.0, the returned result doesn't contain a "bbox" property anymore. It was renamed to "key" to be more along the lines of the mapreduce views. The structure also changes a bit. The "bbox" was an array with 4 elements, with [min_x, min_y, max_x, max_y]. The key is an array containing the bounds of the dimensions, so it is: [[min_x, max_x], [min_x, max_y]].




[JCBC-508] PersistTo.TWO doesn't fail with single node cluster. Created: 11/Aug/14  Updated: 11/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.5
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In the observe poll part, the persist counter gets decremented which leads to a non-failing op on a single node cluster with PersistTo.TWO




[JCBC-488] Getting Failed to Access the View Exception Created: 08/Jul/14  Updated: 07/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.0, 1.4.1
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: meet_ravip Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
Relates to

 Description   
Caused by: java.lang.RuntimeException: Failed to access the view
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:787)


Caused by: java.util.concurrent.ExecutionException: OperationException: GENERAL
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:98)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:780)
        ... 9 more
Caused by: OperationException: GENERAL
        at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:74)
        at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
        at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
        at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
        at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
        at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)


 Comments   
Comment by Michael Nitschinger [ 08/Jul/14 ]
Hi,

can you please give us more information on when you are seeing the exception (what code are you executing, whats going on at the server - nothing, rebalance, failover,..) Thanks!
Comment by meet_ravip [ 08/Jul/14 ]
Hi,
  We use couchbase 1.4 client and its dependent binaries and below is the code snippet. The failure happens in client.query method. We are running 200 concurrent request to perform the below operation. When couchbase running for more than few and if we start this test it fails, after restarting couchbase when this issue happens and restart the test then it works. Our's 3 node cluster setup. At that time failure there is no rebalance or failover. Lets know if you need any further details.
                        

CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
cfb.setOpTimeout(opTimeOut); //100000
cfb.setOpQueueMaxBlockTime(queueMaxBlockTime);30000
cfb.setTimeoutExceptionThreshold(timeoutExceptionThreshold);//998
cfb.setReadBufferSize(readBufSize);//16384
cfb.setShouldOptimize(shouldOptimize);//false
cfb.setMaxReconnectDelay(maxReconnectDelay);//30
cfb.setObsPollInterval(obsPollInterval);//10
cfb.setObsTimeout(obsTimeOut);//5
cfb.setViewTimeout(viewTimeout);//300
cfb.setViewWorkerSize(viewWorkerSize);//1
cfb.setViewConnsPerNode(viewConnectionsPerNode);//100

CouchbaseClient client = new CouchbaseClient(cfb.buildCouchbaseConnection(
hosts, bucketName, userName, password));


Query query = new Query();
query.setReduce(false);
// query.setIncludeDocs(true);
query.setIncludeDocs(false);
query.setStale(Stale.FALSE);


View view = client.getView(designDocumentName, viewName);
if (view != null) {
ViewResponse result = client.query(view, query);
if (result != null) {
List<String> keys = new ArrayList<String>();
for (ViewRow row : result) {
if (row.getKey() != null && !row.getKey().isEmpty()) {

keys.add(row.getKey());
}
}
if (keys.size() > 0) {

documents = client.getBulk(keys);
}
}
}

View Definition:-

function (doc,meta) { emit(meta.id,null);
Comment by Vladislav Koroteev [ 08/Jul/14 ]
I have same issue. I noticed that this exception occured when I do request to view and view is indexing.
Comment by RICHARDS PETER [ 09/Jul/14 ]
In our use case, data update in the couchbase bucket happens only once in a day. During the remaining day, all our clients only read the data from these buckets and still we face this issue. Could you please tell us whether it is possible to control indexing of the view?
Comment by Michael Nitschinger [ 09/Jul/14 ]
Thanks folks,

could you also tell me which server versions you are using? the Error: General points to a 500 response from the server, for whatever reason.
Comment by RICHARDS PETER [ 09/Jul/14 ]
Hi Michael,
We are using 2.1.1 community edition (build-764).
Comment by RICHARDS PETER [ 09/Jul/14 ]
Hi Michael,
Could you also tell me the server log files that should be analyzed to find more details about this error?
Comment by Michael Nitschinger [ 09/Jul/14 ]
Peter,

there is a cbcollect_info command that you can use to aggregate this info, but I would be interested in something else if you can take the time: enable FINEST logging (detailed info here http://nitschinger.at/Logging-with-the-Couchbase-Java-Client) and can you share that log? in there we should have more info from the actual server response and see whats going on.

Also, you could set a breakpoint at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:74) and print the original exception https://github.com/couchbase/couchbase-java-client/blob/release14/src/main/java/com/couchbase/client/protocol/views/ViewOperationImpl.java#L74

maybe we should rethrow that.
Comment by Vladislav Koroteev [ 09/Jul/14 ]
I tried reproduce "java.lang.RuntimeException: Failed to access the view", but I can't. I do workload on Couchbase, so I can observer permanent indexing at web-interface. From JavaSDK I do query with stale=false. Earlier I get "java.lang.RuntimeException: Failed to access the view", now I have

java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:781)
at my.pack.MainView.main(MainView.java:29)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
... 2 more

I tried it on Couchbase Server 2.2 and Couchbase Server 2.5.1 EE, on Java SDK 1.3.2, 1.4.1, 1.4.3.

Maybe this exception occures while view is compacting, not indexing?
Comment by Michael Nitschinger [ 14/Jul/14 ]
I have a change pending that will at least print out the unparsable/unexpected JSON body so it will be easier to debug: http://review.couchbase.org/#/c/39347/
Comment by RICHARDS PETER [ 15/Jul/14 ]
Hi Michael,

Sorry for the delay in giving more details. A system upgrade at our side delayed further testing of this issue. I downloaded the source code of the couchbase client and added some log statements before the new exception is set. The code looks as follows:

************************************************************************************
  @Override
  public void handleResponse(HttpResponse response) {
    String json = getEntityString(response);
    int errorcode = response.getStatusLine().getStatusCode();
    try {
      OperationStatus status = parseViewForStatus(json, errorcode);
      ViewResponse vr = null;
      if (status.isSuccess()) {
        vr = parseResult(json);
      } else {
        parseError(json, errorcode);
      }

      ((ViewCallback) callback).gotData(vr);
      callback.receivedStatus(status);
    } catch (ParseException e) {
System.out.println("ViewOperationImpl handleResponse()"+e.getMessage());
e.printStackTrace();
      LOGGER.log(Level.WARNING,"ViewOperationImpl handleResponse() "+e.getMessage(),e);
setException(new OperationException(OperationErrorType.GENERAL,
          "Error parsing JSON"));
    }
    callback.complete();
  }

************************************************************************************

I was able to reproduce the issue with the new jar file packaged with the above code change. Please find the logs in our application:

2014-07-15 12:11:28 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-15 12:11:28 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-15 12:11:28 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-15 12:11:28 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-15 12:11:28 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)

Please let me know what could be going wrong. The issue does not occur always. It mostly happens after some restarts of our application and only a few clients are getting this issue.

Thanks,
Richards Peter.
Comment by Michael Nitschinger [ 15/Jul/14 ]
Hey, could you do me a favor and print both the HttpResponse and or the raw content body? from the log message I cannot see what actually got returned from the server only that we were unable to parse it.
Comment by RICHARDS PETER [ 16/Jul/14 ]
Hi Michael,

I added more print statements in the catch block to capture the request and response. Please find the modified code and the logs:

 public void handleResponse(HttpResponse response) {
   //retained the same code till here.
    } catch (ParseException e) {
System.out.println("ViewOperationImpl handleResponse()"+e.getMessage());
e.printStackTrace();
System.out.println("ViewOperationImpl handleResponse() HttpRequest: "+getRequest().toString());
System.out.println("ViewOperationImpl handleResponse() HttpResponse: "+response.toString()+" json:"+json+" errorcode:"+ errorcode);
      LOGGER.log(Level.WARNING,"ViewOperationImpl handleResponse() "+e.getMessage(),e);
setException(new OperationException(OperationErrorType.GENERAL,
          "Error parsing JSON"));
    }
    callback.complete();
  }

The issue was reproduced today morning. Please find our application log:

2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-16 10:36:50 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)


2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse()Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] java.text.ParseException: Cannot read json:
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.HttpOperationImpl.parseViewForStatus(HttpOperationImpl.java:140)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:64)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103)
2014-07-16 10:36:50 STDIO [ERROR] at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
2014-07-16 10:36:50 STDIO [ERROR] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
2014-07-16 10:36:50 STDIO [ERROR] at java.lang.Thread.run(Thread.java:744)

2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse() HttpRequest: GET /COMMON_DIMENSION/_design/COMMON_DIMENSION/_view/COMMON_DIMENSION?reduce=false&stale=false [Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Connection: Keep-Alive, User-Agent: JCBC/1.2, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q
2014-07-16 10:36:50 STDIO [INFO] 09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09
2014-07-16 10:36:50 STDIO [INFO] NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NT
2014-07-16 10:36:50 STDIO [INFO] U9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9
2014-07-16 10:36:50 STDIO [INFO] OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm1pub:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: 10.62.81.165:8092, Authorization: Basic Q09NTU9OX0RJTUVOU0lPTjo=, Host: storm2pub:8092]


2014-07-16 10:36:50 STDIO [INFO] ViewOperationImpl handleResponse() HttpResponse: HTTP/1.1 400 Bad Request [Server: MochiWeb/1.0 (Any of you quaids got a smint?), Date: Wed, 16 Jul 2014 05:06:50 GMT, Content-Length: 0] json: errorcode:400
Comment by Michael Nitschinger [ 16/Jul/14 ]
Hi,

thanks for reporting this. I'm not exactly sure whats going on here, could it be that all those authorization headers are part of _one request_?

Can you clarify if you are using the default bucket or something named differently?
I think this is a bug, but I'm not entirely sure if this is the same as the others reported here, let me get a fix up for this since this seems to be trivial.
Comment by RICHARDS PETER [ 16/Jul/14 ]
We have created a new bucket - COMMON_DIMENSION and this bucket has a view also with the same name. The view definition and code to retrieve the data is already shared by meet_ravip.
Comment by Michael Nitschinger [ 16/Jul/14 ]
Thanks, the good news is that I have a trivial bugfix for your case I think. Currently we retry a request if it fails for some reason against a node, but the code just adds the headers again and I guess the server just rejects it at some point. So I'll add checks around the add to make sure they are only added once.
Comment by Michael Nitschinger [ 16/Jul/14 ]
See: http://review.couchbase.org/#/c/39429/

I think this will get into 1.4.4, but you can patch it on your own if you can't wait for it. If you do, let me know if it works
Comment by RICHARDS PETER [ 16/Jul/14 ]
Thanks Michael,
That is really a good news. Please share the details of the fix when it is available. We will test the fix in our setup and update you about our findings.
Comment by Michael Nitschinger [ 16/Jul/14 ]
1.4.4 is scheduled for first week of august, but I'll merge it into master much sooner (this week I hope) once it goes through review. If you know how to build it from github (if not I can help you with) you can play around with it sooner.
Comment by RICHARDS PETER [ 16/Jul/14 ]
Thanks Michael.
I am about to test the aforementioned patch on my setup. I have applied the changes to the 1.4 source code which we downloaded a few days back. I will update you about our observation. Since this issue cannot be reproduced very quickly, my update on this issue may be slightly delayed (24 hours minimum).
Comment by RICHARDS PETER [ 17/Jul/14 ]
Hi Michael,
I would like to clarify one more doubt related to this comment:
"Currently we retry a request if it fails for some reason against a node, but the code just adds the headers again and I guess the server just rejects it at some point. "
We would like to minimize such failures. Is it possible to find out the reason why the request failed against a node? Is it possible to capture such cases in our log by modifying any class? Could you please guide us on the same?
Comment by Michael Nitschinger [ 17/Jul/14 ]
Hi,

normally you don't need to be concerned about that. There are various situations as part of the regular operations (like adding a node) where there is a limited timespan where the node is added, but does not contain any partitions yet. We get a response from the server and retry behind the scenes. Also with failovers where the config is not updated the node returns an error and we retry.

If we don't get to a good result we will try until the op times out with the given timeout, so you should be good on that front. Of course, the behavior reported here was because of a bug.
Comment by RICHARDS PETER [ 17/Jul/14 ]
Thanks Michael,
We normally get this error after some restart of our application. Till now we were not able to reproduce this issue. However we got a new error after the latest restart of our application. The stacktrace is as follows:

Caused by: com.couchbase.client.vbucket.ConfigurationException: Could not fetch a valid Bucket configuration.
at com.couchbase.client.vbucket.provider.BucketConfigurationProvider.bootstrap(BucketConfigurationProvider.java:123)
at com.couchbase.client.vbucket.provider.BucketConfigurationProvider.getConfig(BucketConfigurationProvider.java:373)
at com.couchbase.client.CouchbaseConnectionFactory.getVBucketConfig(CouchbaseConnectionFactory.java:317)
at com.couchbase.client.CouchbaseClient.(CouchbaseClient.java:258)

All that we did was an application restart. There were no other changes in the environment. We could see all the buckets on the couchbase nodes when this error was produced. The buckets also had the same items as before. During the restart there will be around 200 threads accessing the same data. I would like to know the possible scenarios which can lead to this error. Would you like me to open a new issue for this?
Comment by Michael Nitschinger [ 17/Jul/14 ]
Yes please, and if you could provide debug level logging (or finest even) would be good, thanks!




[JCBC-503] Client shutdown doesn't release observers and leak memory. Created: 29/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: Benoit Beaudet Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If a client is using VBucket the shutdown method call BucketConfigurationProvider.shutdown().

{code}
try {
      CouchbaseConnectionFactory cf = (CouchbaseConnectionFactory) connFactory;
      cf.getConfigurationProvider().shutdown();
      if(vconn != null) {
        vconn.shutdown();
      }
{code}

The observers array reference closed CoucheBaseClient.
To fix the issue, I need to call directly unsubscribe method.
connectionFactory.getConfigurationProvider().unsubscribe((Reconfigurable) client);
 

 Comments   
Comment by Michael Nitschinger [ 04/Aug/14 ]
Does that look valid?
http://review.couchbase.org/#/c/40263/
Clearing them on shutdown seems valuable
Comment by Benoit Beaudet [ 04/Aug/14 ]
Hi!

I think that the observers list is global. A shutdown of 1 client will clears all active clients observers?.
This is why I used unsubscribe.

 @Override
  public void unsubscribe(Reconfigurable rec) {
    observers.remove(rec);
  }
Comment by Michael Nitschinger [ 05/Aug/14 ]
Hey,

the thing is if you shut down the config manager noone else can use it anyway because its down, so we can just unsubscribe all of them right in the shutdown method. There is no need to keep more of them around, especially because this is intended to be internal interface.
Comment by Benoit Beaudet [ 05/Aug/14 ]
In my scenario. I use a pool to create more connections under load and release them after an idle period.
I shutdown clients not only at web app shutdown, but also at runtime.

The factory is a singleton in my project. For each CouchbaseClientConnection created with the factory, an observer is added.
cf.getConfigurationProvider().subscribe(this);

CouchbaseClient.shutdown method call indirectly the factory getConfigurationProvider().
   CouchbaseConnectionFactory cf = (CouchbaseConnectionFactory) connFactory;
      cf.getConfigurationProvider().shutdown();

This is why is suggested that observers array size should match created connections. Using internal subscribe/unsubscribe to match the pattern.



Comment by Michael Nitschinger [ 05/Aug/14 ]
Every time you create a new CouchbaseClient object, all of those objects are created again.

You should never ever shut down or listen on the configuration provider on your own, that is internal behavior. You should 1) always only use the CouchbaseClient interface and 2) reuse them as long as possible.

Why do you shut them down at runtime?
Comment by Benoit Beaudet [ 05/Aug/14 ]
I use CouchbaseClient.shutdown() to evict clients and use new CouchbaseClient(using the singleton CouchbaseConnectionFactory as parameter) to recreate them. I shouldn't use the Configuration provider directly.
But the internal configuration provider is keeping references on all connections created, even if the pool closed them. I create more than 1 client at bootup and close it at shutdown.
The pool manage the growth and idling clients. It's the reason to shutdown a client if idling for a long time.




[JCBC-483] Add INFO level log of client version to startup Created: 27/Jun/14  Updated: 05/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.2
Fix Version/s: 1.4.5
Security Level: Public

Type: Improvement Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In CouchbaseClient <init> we log details from the CouchbaseConnectionFactory. It would also be useful to INFO level log the client version and the spymemcached version.

 Comments   
Comment by Matt Ingenthron [ 27/Jun/14 ]
Since it's such a minor change, setting it to 1.4.3. I'll defer to you on whether it should be pushed back or not.
Comment by Michael Nitschinger [ 30/Jun/14 ]
It's a little harder to do because there is the BuildInfo, but we need to make sure its properly in path and distributed with our jars. I'm not sure this is done right now but I could be wrong.

If we could defer it to 1.4.4 then it would be better to release 1.4.3 tomorrow.




[JCBC-464] Client never recovers when all nodes are shut down Created: 03/Jun/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File thread-dump.txt    
Issue Links:
Relates to

 Description   
When running this demo, if I disconnect the cluster entirely and restart it, the client reconnects but never recovers:
https://github.com/couchbaselabs/CBLiveDemo

 Comments   
Comment by Michael Nitschinger [ 14/Jul/14 ]
http://review.couchbase.org/#/c/39346/




[JCBC-490] Log JSON body when failed to parse valid body Created: 14/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This change does not fix an error itself, but makes it much easier to discover what valid JSON gets sent over where we are failing to handle it.




[JCBC-505] Concurrency issue in the View Query builder with regexes Created: 30/Jul/14  Updated: 05/Aug/14  Resolved: 05/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-495] 1.4x doesn't build if source dir contains whitespace Created: 16/Jul/14  Updated: 04/Aug/14  Resolved: 04/Aug/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.3
Fix Version/s: 1.4.4
Security Level: Public

Type: Bug Priority: Minor
Reporter: David Haikney Assignee: David Haikney
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
After running a git clone; git checkout of 1.4.3 I was unable to build on my local machine. "ant package" yielded the following:

 [exec] /Users/dhaikney/Documents/Google Drive/dev/couchbase-java-client/src/scripts/write-version-info.sh: line 47: ${changesfile}: ambiguous redirect

Some of the variables in the write-version-info.sh script need surrounding quotes to allow for the whitespace.

 Comments   
Comment by David Haikney [ 16/Jul/14 ]
http://review.couchbase.org/39443




[JCBC-437] implement JSR-107 Java Temporary Caching API Created: 25/Mar/14  Updated: 31/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Task Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
As previously discussed, we should look to implement JSR-107 to meet the standards.




[JCBC-501] Implement Support for DCP Created: 29/Jul/14  Updated: 29/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: New Feature Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Implement support for 3.0+ DCP as a replacement for the TAP protocol and TAP Client.






[JCBC-496] Add unit test for JCBC-488 Created: 22/Jul/14  Updated: 28/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .next
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Wei-Li Liu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We need to add a unit test for the fix in http://www.couchbase.com/issues/browse/JCBC-488 just to make sure it works as expected. The appropriate change ishttp://review.couchbase.org/#/c/39429/ and has been merged already.

I'd propose a unit test which mocks out the underlying stuff and just pushes a HttpOperation a few times through the ViewConnection#addOp which should add the headers only 1 times instead of N.

 Comments   
Comment by Wei-Li Liu [ 25/Jul/14 ]
I think we have something similar to JCBC-488 test case in the scenario test framework (Service Restart)

I pull in your fix http://review.couchbase.org/#/c/39429/ and reproduce the issue by switching thread # to 200 instead of 10 (default)
The report shows that if thread is 10 works for both mc (get,set) and ht, cb (view, query). If jumps thread # to 200, view and query workload would fail
https://docs.google.com/spreadsheets/d/1sjDpJL_xuVv3kDhbnJKn5qNLmvzCi80gbdq5FkOObzQ/edit#gid=1476387871

I can make a separate unit test to make sure when there are multiple http operations, there can only be 1 header.






[JCBC-151] Client does timeout on connect in specific java environments (was believed to be java7 related). Created: 21/Nov/12  Updated: 21/Jul/14  Resolved: 31/Dec/13

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.1-dp4
Fix Version/s: 1.3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to JCBC-388 Refactor View Connection Management Resolved

 Description   
People report that the Client does not work with 1.7. Here is a sample stack trace:

2012-11-20 00:29:11.228 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-11-20 00:29:11.240 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@3f757322
2012-11-20 00:29:11.502 INFO com.couchbase.client.ViewConnection: Added localhost/127.0.0.1:8092 to connect queue
2012-11-20 00:29:11.505 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2012-11-20 00:29:11.647 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210
2012-11-20 00:29:12.051 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2012-11-20 00:29:12.060 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: localhost/127.0.0.1:8092 - Session request successful
2012-11-20 00:29:17.111 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [localhost/127.0.0.1:8092(closed)]
java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:68)
at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:428)
at Example1.main(Example1.java:43)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:80)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:65)
... 2 more
2012-11-20 00:30:12.168 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
2012-11-20 00:30:12.177 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
Disconnected from the target VM, address: '127.0.0.1:65280', transport: 'socket'

Process finished with exit code 0


Also See http://www.couchbase.com/forums/thread/java-asyncconnectionmanager-timed-out-waiting-operation-please-help-console-log-included
And http://stackoverflow.com/questions/13466010/using-java-api-in-scala-to-query-views-in-couchbase-throws-timeout-exception?utm_source=twitterfeed&utm_medium=twitter

 Comments   
Comment by Quinn Slack [ 27/Nov/12 ]
This doesn't appear to be strictly related to Java7. I was able to get similar code to work on Java7. I even simulated loading a Play2 environment with other JARs that could potentially cause conflicts. It's possible that my simulation of the Play2 environment was insufficient and that Play2 does other stuff...

Code at https://github.com/sqs/couchbase-scala-example. Run with:

CBURL=http://localhost:8091/pools CBPASSWORD=mypassword sbt -Dconfig.file=conf/application.conf '~run'

Here is the output on my system that shows it's on Java7 (openjdk) and that shows the expected output from the views:

on java version 1.7.0_09
[info] play - Application started (Prod)
2012-11-27 01:34:27.115 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2012-11-27 01:34:27.120 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@6267e5a2
2012-11-27 01:34:27.147 INFO com.couchbase.client.ViewConnection: Added localhost/127.0.0.1:8092 to connect queue
2012-11-27 01:34:27.149 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2012-11-27 01:34:27.176 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210
2012-11-27 01:34:27.282 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2012-11-27 01:34:27.288 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: localhost/127.0.0.1:8092 - Session request successful
Res = List(com.couchbase.client.protocol.views.ViewRowNoDocs@4667820f, com.couchbase.client.protocol.views.ViewRowNoDocs@358bcae5, com.couchbase.client.protocol.views.ViewRowNoDocs@6cb59bd9, com.couchbase.client.protocol.views.ViewRowNoDocs@70afb51, com.couchbase.client.protocol.views.ViewRowNoDocs@61f98673)
2012-11-27 01:34:27.396 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client
2012-11-27 01:34:27.403 INFO com.couchbase.client.ViewNode: Couchbase I/O reactor terminated
Comment by Michael Nitschinger [ 27/Nov/12 ]
Are you running on jdk7 on mac? Also, when you use play 2.1 it should work. Play2-related issues more were because of netty version incompatibilities.
Comment by Quinn Slack [ 27/Nov/12 ]
This is jdk7 on Arch Linux and Play 2.1-RC1.

I tried using a different netty, but no luck. What worked for me was moving calls to "new CouchbaseClient" out of static "object" initializers, since they appeared to be getting called in the Netty I/O thread (I got this exception in new CouchbaseClient: java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread).
Comment by Michael Nitschinger [ 28/Nov/12 ]
Hi Quinn, this issue is a different one! The issue described here is that people seem to find connection timeouts with java 7.
Comment by Quinn Slack [ 28/Nov/12 ]
They may be related--or may just have similar symptoms. I was not getting timeout exceptions, but I was seeing view requests hang indefinitely until I made the fix described above.
Comment by dragos [ 13/Mar/13 ]
Hi Michael, can you help with this error ? I am trying to find a workaround or something to work. This is the only thing which is keeping us from using Couchbase. I also posted here : http://www.couchbase.com/forums/thread/couchbase-connectivity-problem-aws-vpc . As you can see in the times the timeout is received really fast.

2013-03-13 11:40:47.702 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.X.XXX:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-03-13 11:40:52.712 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@5cbfe9d
2013-03-13 11:40:52.840 INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.X.XXX/10.0.X.XXX:11210
2013-03-13 11:40:57.860 INFO com.couchbase.client.ViewConnection: Added 10.0.X.XXX to connect queue
2013-03-13 11:40:57.863 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-03-13 11:40:58.070 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-03-13 11:40:58.077 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.X.XXX:8092 - Session request successful
2013-03-13 11:41:03.113 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.X.XXX/10.0.X.XXX:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
        at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:475)
        at couchbase.training.view.poc.CouchbasePOC.main(CouchbasePOC.java:63)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi dragos,

can you please post the full code where you connect to couchbase? (including factories and such)

Thanks!
Comment by dragos [ 13/Mar/13 ]
Hi Mike, this is the POC client for Couchbase:
package couchbase.training.view.poc;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.net.URI;
import java.util.LinkedList;
import java.util.List;
import java.util.concurrent.TimeUnit;

import org.apache.commons.io.IOUtils;
import org.json.JSONException;
import org.json.XML;

import com.couchbase.client.CouchbaseClient;
import com.couchbase.client.CouchbaseConnectionFactory;
import com.couchbase.client.CouchbaseConnectionFactoryBuilder;
import com.couchbase.client.protocol.views.ComplexKey;
import com.couchbase.client.protocol.views.Query;
import com.couchbase.client.protocol.views.Stale;
import com.couchbase.client.protocol.views.View;
import com.couchbase.client.protocol.views.ViewResponse;
import com.couchbase.client.protocol.views.ViewRow;

public class CouchbasePOC
{
public static void main(String[] args) throws FileNotFoundException, JSONException, IOException
{
System.out.println("Hello from CouchBase");

// Set the URIs and get a client
final List<URI> uris = new LinkedList<URI>();

// Connect to localhost or to the appropriate URI(s)
uris.add(URI.create("http://10.0.7.164:8091/pools"));

final String mappedName="problems";
long start = System.currentTimeMillis();

CouchbaseConnectionFactoryBuilder m_couchbaseConnectionFactoryBuilder = new CouchbaseConnectionFactoryBuilder();
m_couchbaseConnectionFactoryBuilder.setMaxReconnectDelay(10000);
m_couchbaseConnectionFactoryBuilder.setOpQueueMaxBlockTime(100);
m_couchbaseConnectionFactoryBuilder.setOpTimeout(20000);
m_couchbaseConnectionFactoryBuilder.setShouldOptimize(true);
m_couchbaseConnectionFactoryBuilder.setViewTimeout(300000);

CouchbaseConnectionFactory connFactory = m_couchbaseConnectionFactoryBuilder.buildCouchbaseConnection(uris, mappedName, "");

CouchbaseClient client = null;
try
{
client = new CouchbaseClient(connFactory);
}
catch (IOException e)
{
System.err.println("IOException connecting to Couchbase: " + e.getMessage());
System.exit(1);
}

final View view = client.getView(mappedName, mappedName);
final Query query = new Query();

final ComplexKey key = ComplexKey.of("problem",null,null,null,null);
query.setKey(key);
query.setStale( Stale.UPDATE_AFTER );
int counter=0;
try
{
final ViewResponse result = client.query(view, query);
for(ViewRow row : result)
{
counter++;
System.out.println(row.getId());
}
}
catch(java.lang.VerifyError e)
{
System.out.println("Error: " + e.getMessage());
}

System.out.println("time: " + (System.currentTimeMillis()-start) );
System.out.println("documents: " + counter );

client.shutdown(3, TimeUnit.SECONDS);
System.exit(0);
}
}
Comment by dragos [ 13/Mar/13 ]
I hope you would not mind that i called you Mike and not Michael.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hehe no worries.

Can you do me a favor and remove the factory for a second and just use CouchbaseClient() without anything else? Let me know if the problem still persists or not! Thanks
Comment by dragos [ 13/Mar/13 ]
Hi Michael,
First i want to thank you for the quick feed-back. Your owesome !!!

Before using the factory i used the CouchbaseClient(uris, bucketName, "") which is also timing out. I added the factory to be able increase the timeout times for the connection but still the same problem.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hmm okay, so its not the factory (we had some issues with default values in the past, thats why I'm asking).

Can you pin me down your operating system and exact java version you're using?Thanks for your help!
Comment by Michael Nitschinger [ 13/Mar/13 ]
Also, since you're so responsive (I hope we can pin this down finally!).. Can you run the same code with DEBUG log enabled (see https://code.google.com/p/spymemcached/wiki/Logging).. just use Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST); instead of Logger.getLogger("net.spy.memcached").setLevel(Level.FINEST);

Thanks!
Comment by dragos [ 13/Mar/13 ]
This is my test environment for the client which will be similar for production.
OS Ubuntu Server 11.10 x64
java -version result:
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My network infrastructure is like this:
In an AWS virtual private center (VPC) i have a subnet and in this subnet there are 2 machines. The client machine and the Couchbase server. The machines have identical OS and java version as mentioned above. I have open the ports and also checked the network ACL. The traffic is free to go,no restriction. If i am using a rest query with curl for the view i receive the result, the problem is only in the java SDK.
If needed more information please ask. I will gladly offer it.

Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi Dragos,

yes, if you can please rerun the script with DEBUG logging enabled (see last comment).. Thanks very much in advance!
Comment by dragos [ 13/Mar/13 ]
I've setted the logger to finest and this is the error log obtained. Also i tested again the connection with curl and no problemes.

Mar 13, 2013 1:30:14 PM com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
2013-03-13 13:30:14.679 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-03-13 13:30:19.689 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@4e99353f
2013-03-13 13:30:19.810 INFO net.spy.memcached.auth.AuthThread: Authenticated to 10.0.7.164/10.0.7.164:11210
2013-03-13 13:30:24.828 INFO com.couchbase.client.ViewConnection: Added 10.0.7.164 to connect queue
2013-03-13 13:30:24.830 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-03-13 13:30:25.025 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-03-13 13:30:25.033 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /10.0.7.164:8092 - Session request successful
2013-03-13 13:30:30.083 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
        at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:475)
        at couchbase.training.view.poc.CouchbasePOC.main(CouchbasePOC.java:68)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
        at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
        at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)
        ... 2 more

curl result

curl http://10.0.7.164:8092/problems/_design/dev_problems/_view/problems?stale=false\&connection_timeout=60000\&limit=10\&skip=0
{"total_rows":0,"rows":[
]
}
[4]+ Done
Comment by dragos [ 13/Mar/13 ]
I am available for next steps in debuging. With what can i help next ?
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi, hmm no there should be more output (DEBUG), not just info... you need to copy this whole snippet before your code actually executes:

        // Tell spy to use the SunLogger
        Properties systemProperties = System.getProperties();
        systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");
        System.setProperties(systemProperties);

        Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);

        //get the top Logger
        Logger topLogger = java.util.logging.Logger.getLogger("");

        // Handler for console (reuse it if it already exists)
        Handler consoleHandler = null;
        //see if there is already a console handler
        for (Handler handler : topLogger.getHandlers()) {
            if (handler instanceof ConsoleHandler) {
                //found the console handler
                consoleHandler = handler;
                break;
            }
        }

        if (consoleHandler == null) {
            //there was no console handler found, create a new one
            consoleHandler = new ConsoleHandler();
            topLogger.addHandler(consoleHandler);
        }

        //set the console handler to fine:
        consoleHandler.setLevel(java.util.logging.Level.FINEST);


I guess you're running it from your IDE?
Comment by dragos [ 13/Mar/13 ]
I am building the client on local IDE then uploading it to EC2 test machine. The snipped of code word and now i have a full log.

Mar 13, 2013 1:53:10 PM com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools/default?uuid=44b1e62dd7e65cd38ae7faaabe5ebb64
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP readToString
FINE: Attempting to read configuration from URI: http://10.0.7.164:8091/pools/default/buckets?v=120523822&uuid=44b1e62dd7e65cd38ae7faaabe5ebb64
Mar 13, 2013 1:53:11 PM net.spy.memcached.MemcachedConnection createConnections
INFO: Added {QA sa=/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Mar 13, 2013 1:53:11 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Updating nodesMap in VBucketNodeLocator.
Mar 13, 2013 1:53:16 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Adding node with address 10.0.7.164:11210.
Mar 13, 2013 1:53:16 PM com.couchbase.client.vbucket.VBucketNodeLocator fillNodesEntries
FINE: Node added is {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=8}.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=false, w=false, c=true, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=8})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@2c76e369
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection insertOperation
FINE: Added Cmd: 10 Opaque: 1 to {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=0, #iq=1, topRop=null, topWop=null, toWrite=0, interested=8}
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Read 24 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Completed read op: Cmd: 10 Opaque: 1 and giving the next 0 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleInputQueue
FINE: Handling queue
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: No selectors ready, interrupted: false
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: No selectors ready, interrupted: false
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleInputQueue
FINE: Handling queue
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection insertOperation
FINE: Added SASL auth operation to {QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL auth operation, toWrite=0, interested=4}
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=false, w=true, c=false, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL auth operation, toWrite=0, interested=4})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selected 1, selected 1 keys
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Handling IO for: sun.nio.ch.SelectionKeyImpl@2c76e369 (r=true, w=false, c=false, op={QA sa=10.0.7.164/10.0.7.164:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL auth operation, topWop=null, toWrite=0, interested=1})
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Read 37 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.auth.AuthThread$1 receivedStatus
INFO: Authenticated to 10.0.7.164/10.0.7.164:11210
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleReads
FINE: Completed read op: SASL auth operation and giving the next 0 bytes
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Done dealing with queue.
Mar 13, 2013 1:53:16 PM net.spy.memcached.MemcachedConnection handleIO
FINE: Selecting with delay of 0ms
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewConnection createConnections
INFO: Added 10.0.7.164 to connect queue
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseClient <init>
INFO: viewmode property isn't defined. Setting viewmode to production mode
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ConfigurationProviderHTTP subscribe
FINE: Subscribing an object for reconfiguration updates com.couchbase.client.CouchbaseClient
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686] OPEN


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is true and Channel State is OPEN.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686, /10.0.7.213:55729 => /10.0.7.164:8091] BOUND: /10.0.7.213:55729


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is /10.0.7.213:55729 and Channel State is BOUND.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINEST: Channel state changed: [id: 0x2e273686, /10.0.7.213:55729 => /10.0.7.164:8091] CONNECTED: /10.0.7.164:8091


Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler handleUpstream
FINER: Channel state change is not a disconnect. Event value is /10.0.7.164:8091 and Channel State is CONNECTED.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: STATUS: 200 OK
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: VERSION: HTTP/1.1
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Cache-Control = no-cache
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Content-Type = application/json; charset=utf-8
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Date = Wed, 13 Mar 2013 13:55:42 GMT
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Pragma = no-cache
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Server = Couchbase Server 2.0.0-1976-rel-enterprise
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: HEADER: Transfer-Encoding = chunked
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER:

Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: CHUNKED CONTENT {
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: {"name":"problems","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","streamingUri":"/pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","localRandomKeyUri":"/pools/default/buckets/problems/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/problems/controller/compactBucket","compactDB":"/pools/default/buckets/problems/controller/compactDatabases"},"nodes":[{"couchApiBase":"http://10.0.7.164:8092/problems","replication":0.0,"clusterMembership":"active","status":"healthy","thisNode":true,"hostname":"10.0.7.164:8091","clusterCompatibility":131072,"version":"2.0.0-1976-rel-enterprise","os":"x86_64-unknown-linux-gnu","ports":{"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/problems/stats","directoryURI":"/pools/default/buckets/problems/statsDirectory","nodeStatsListURI":"/pools/default/buckets/problems/nodes"},"ddocs":{"uri":"/pools/default/buckets/problems/ddocs"},"nodeLocator":"vbucket","autoCompactionSettings":false,"fastWarmupSettings":false,"uuid":"7675b7791efe17220792c2b41ff6c824","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.0.7.164:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 3066
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: 0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 2048
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: ],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 3072
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: 1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"bucketCapabilitiesVer":"","bucketCapabilities":["touch","couchapi"]}
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketUpdateResponseHandler finerLog
FINER: Chunk length is: 361
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.BucketMonitor logFiner
FINER: Getting server list returns this last chunked response:
{"name":"problems","bucketType":"membase","authType":"sasl","saslPassword":"","proxyPort":0,"replicaIndex":false,"uri":"/pools/default/buckets/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","streamingUri":"/pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824","localRandomKeyUri":"/pools/default/buckets/problems/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/problems/controller/compactBucket","compactDB":"/pools/default/buckets/problems/controller/compactDatabases"},"nodes":[{"couchApiBase":"http://10.0.7.164:8092/problems","replication":0.0,"clusterMembership":"active","status":"healthy","thisNode":true,"hostname":"10.0.7.164:8091","clusterCompatibility":131072,"version":"2.0.0-1976-rel-enterprise","os":"x86_64-unknown-linux-gnu","ports":{"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/default/buckets/problems/stats","directoryURI":"/pools/default/buckets/problems/statsDirectory","nodeStatsListURI":"/pools/default/buckets/problems/nodes"},"ddocs":{"uri":"/pools/default/buckets/problems/ddocs"},"nodeLocator":"vbucket","autoCompactionSettings":false,"fastWarmupSettings":false,"uuid":"7675b7791efe17220792c2b41ff6c824","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.0.7.164:11210"],"vBucketMap":[[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1],[0,-1]]},"bucketCapabilitiesVer":"","bucketCapabilities":["touch","couchapi"]}
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: Received an update, notifying reconfigurables about a com.couchbase.client.vbucket.config.Bucketcom.couchbase.client.vbucket.config.Bucket@4cf3fdc4
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: It says it is problems and it's talking to /pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseConnection reconfigure
FINE: Node 10.0.7.164/10.0.7.164:11210 will stay in cluster config after reconfiguration.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.VBucketNodeLocator updateLocator
FINE: Received updated configuration with insignificant changes.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: Received an update, notifying reconfigurables about a com.couchbase.client.vbucket.config.Bucketcom.couchbase.client.vbucket.config.Bucket@fadb83cf
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.ReconfigurableObserver update
FINEST: It says it is problems and it's talking to /pools/default/bucketsStreaming/problems?bucket_uuid=7675b7791efe17220792c2b41ff6c824
Mar 13, 2013 1:53:21 PM com.couchbase.client.CouchbaseConnection reconfigure
FINE: Node 10.0.7.164/10.0.7.164:11210 will stay in cluster config after reconfiguration.
Mar 13, 2013 1:53:21 PM com.couchbase.client.vbucket.VBucketNodeLocator updateLocator
FINE: Received updated configuration with insignificant changes.
Mar 13, 2013 1:53:21 PM com.couchbase.client.http.AsyncConnectionManager processConnectionRequests
INFO: Opening new Couchbase HTTP connection
Mar 13, 2013 1:53:21 PM com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback completed
INFO: /10.0.7.164:8092 - Session request successful
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewNode$EventLogger connectionOpen
FINE: Connection open: [/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionTimeout
SEVERE: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionClosed
FINE: Connection closed: [10.0.7.164/10.0.7.164:8092(closed)]
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi, thanks for the log..

interestingly, there is nothing unusual in this.. I'll investigate further and come back to you if I need more info..

thanks so far!
Comment by dragos [ 13/Mar/13 ]
Thank you for spending time on this. Looking forward for a fix :) .
Comment by Michael Nitschinger [ 13/Mar/13 ]
interestingly, the connection is open and then times out...

INFO: /10.0.7.164:8092 - Session request successful
Mar 13, 2013 1:53:21 PM com.couchbase.client.ViewNode$EventLogger connectionOpen
FINE: Connection open: [/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionTimeout
SEVERE: Connection timed out: [10.0.7.164/10.0.7.164:8092]
Mar 13, 2013 1:53:26 PM com.couchbase.client.ViewNode$EventLogger connectionClosed
FINE: Connection closed: [10.0.7.164/10.0.7.164:8092(closed)]

this sounds highly suspicious to me, maybe this is a bug in apache httpcore-nio...
Comment by dragos [ 13/Mar/13 ]
The time out is displayed after 5 seconds. As you suggested i removed the factory and the detailed logged above i obtained with using CouchbaseClient(uris, mappedName, ""). Do you have some way of testing the httpCore-nio ? Should i try to look for latest version of httpCore-nio.. ?
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi,

yes you could try that.. I ran the test suite with 4.2.3 which is the latest one and there were no errors, so our code should work with it. You need to exchange the httpcore and httpcore-nio with the latest 4.2.3 jars

I don't think this is 100% the issue, but it helps us get rid of one more factor!

Thanks!
Michael
Comment by Michael Nitschinger [ 13/Mar/13 ]
Okay, the 5 second period can come from this:

 HttpParams params = new SyncBasicHttpParams();
      params.setIntParameter(CoreConnectionPNames.SO_TIMEOUT, 5000)
          .setIntParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 5000)
....


SO_TIMEOUT is: Defines the socket timeout (SO_TIMEOUT) in milliseconds, which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets).

and CONNECTION_TIMEOUT is: Determines the timeout in milliseconds until a connection is established.


Dragos, can it be the case that it takes more than 5 seconds to return a result when using curl or so? That would explain the "fail fast" timeout here.. not saying that these times are okay this way, just to find the root cause..

Thanks!
Comment by dragos [ 13/Mar/13 ]
Hi Michael,
I've made a test with httpcore-nio-4.2.jar and with httpcore-4.2.1.jar and your intuition is good. The httpcore is not the problem because the timeout is still there. With curl the results is back fast, in less then half a second. So the problem must be somewhere in the java SDK in the way the view is retrieved.
Comment by dragos [ 13/Mar/13 ]
I've moved the Couchbase server to the same machine as the client to have all in same environment for testing. Now all is working fine but it is not possible for our product to stay on the same machine as Couchbase server for obvious reasons. In production we will have more then one app machine which we will try to access the Couchbase cluster.
Comment by Michael Nitschinger [ 13/Mar/13 ]
Hi Dragos,

okay so its definitely something with those timeouts.. The thing is we can increase them for sure, but the question is if it would work for your application given the long network latency - and if your application stays performant that way.

When you're running curl on a request, how long does it take?
Comment by dragos [ 14/Mar/13 ]
The curl request is very fast (a few milliseconds, and less then 0.5 seconds i am sure) and the network latency it is not a problem. Some how the couchbase client is taking to much time connecting. When it need to connect to a network machine in the same lan it takes more then 10 seconds. I experimented this in EC2 environment and on my local machine with a VM. If couchbase is local with the test app, the couchbase client is connecting fast.
Do you have any points on how a connection should be setup for couchbase client ? I am trying to use the factory builder and the connection factory from java sdk but still with no success.
Comment by Tim Smith (Inactive) [ 14/Mar/13 ]
Dragos,

I can think of two separate things to try. One is to increase the SO_TIMEOUT to 8000, CONNECTION_TIMEOUT to 12000. Identify what's being hit.

Probably more effective would be to use Wireshark to capture all traffic on the client machine, and see what is really happening. Start the capture (with tshark command-line tool, for example), then run the test client, let it time out, then run the curl command, let it succeed, then stop the capture. Gzip the capture data and attach it to this issue.

Tim
Comment by dragos [ 14/Mar/13 ]
Hi Tim,
I've started some load tests to be able to evaluate Couchbase, so this means i moved Couchbase server on the same machine as the load test client. I tried to use a pool of couchbase clients always connected to the server, and after a while the test failed because couldn't create more connections. From my initial research i seen that couchbase client is managing his own connection can you tell me how this is done or what are the policies of shutting down connections ? Can you point me to some best practices for couchbase client connectivity ?
Comment by Tim Smith (Inactive) [ 14/Mar/13 ]
I will handle that question about connection pooling, etc., separately, since it is unrelated to this bug about timeouts.

On this specific bug report, the current state as I understand it is that: curl returns very quickly with the correct results. The Java client times out after 5 seconds without returning the results.

My suggestion is to set up a very simple Java client that performs a single view query, and to get a packet capture of both curl and the Java client, to identify what the two are doing differently and why the curl succeeds but the Java client fails.
Comment by Jahangir Zinedine [ 27/Mar/13 ]
Hi Michael,

Any update on this issue?
I faced the same issue and here is my environment:
JRuby 1.7.3
Oracle JDK 1.7(with 1.6 the same thing happened)
Couchbase 2.0.1
Java SDK 1.1.3(with 1.1.4 the same thing happened)
And latest netty and http-core and other jar files.

Appreciate your help.

Thanks,
Jani
Comment by dragos [ 01/Apr/13 ]
Hi All,
Finally after a long time i found the workaround needed for the java client to not timeout. As has been suggested by the Couchbase team to use a host name for the Couchabase server i applied the same fix but to the client machine. Meaning on the client machine were the java Couchbase client was timing out ( in Amazon VPC ) we added a hostname for the Couchbase server. Our test URI from http://10.0.7.164:8091/pools has become http://couchbase:8091/pools. This workaround is also fixing a 10 seconds connectivity for the client.

On Ubuntu the hostname file is at: /etc/hosts
On Windows 7 the hostname file is at: C:\Windows\system32\drivers\etc\hosts

Example of host name entry in client machine:
10.0.7.146 couchbase
Comment by Michael Nitschinger [ 02/Apr/13 ]
Thanks very much dragos for tracking this down!

Does this mean that in your environment after the change, you're not seeing any performance impact anymore? Everything works as expected and performant?

Thanks!
Comment by dragos [ 02/Apr/13 ]
Hi Michael,
In this moment the client does not time out and the time taken by java couchbase client to connect takes around 1.5 seconds from 10 seconds which we can consider a normal behavior. As for performance with this workaround we can continue the POC and evaluate the performance. As a first impression i can say that we see a good speed but i will keep my reservation until final POC is tested and we can achieve a full load test.

What i want to emphasize that this is a welcomed workaround but it is not a solution on a longer term for our cloud production site. In a scenario where we need to create new machines and maybe new couchbase clusters it involves manual or specific automatic changes to the hosts file and of course extra managing of this. Bottom line is that we are waiting the fix on the couchbase java sdk 1.1.5 which will make this workaround obsolete.

Also i feel the need to mention that this workaround was found in collaboration with the Couchbase team which provided the basic information for the workaround to be found, so thank you guys.
Comment by Michael Nitschinger [ 02/Apr/13 ]
Hi Dragos,

I'm not sure if there is a thing that we can fix inside the client when this is a DNS/hostname resolution issue with the OS and amazon?

We would definitely need to investigate further, but all that the client does is try to use the hostname/IP provided during bootstrap. If nothing comes back or the host can't be found, we can't do much about it. Anyway, I think we need to do some more investigation here to really see whats the problem.
Comment by aboyev [ 09/Sep/13 ]
Hello, Michael!
I have the same problem. Here is my setup:
- Couchbase 2.1.1
- Java SDK 1.1.9
- Java 1.6.0_27 (and also tried 1.7)
[ERROR] getView | Exception while doing getView: Timed out waiting for operation
[ERROR] Exception while doing getView: I/O reactor has been shut down
It surely has something to do with the client, because when I launch second client it works properly for a while.
Also, it seems to happen under high loads more frequently
Comment by gaurang jhawar [ 24/Oct/13 ]
How did u change the "Meaning on the client machine were the java Couchbase client was timing out ( in Amazon VPC ) we added a hostname for the Couchbase server. Our test URI from http://10.0.7.164:8091/pools has become http://couchbase:8091/pools. This workaround is also fixing a 10 seconds connectivity for the client." in AWS .. I have EC2 instance for which i want to set the hostname and connect through it like you did... Can you tell me the exact steps for that
Comment by Michael Nitschinger [ 25/Oct/13 ]
Hi Gaurang,

are you also exhibiting a delay on connect? Or during operations? I think these two may be related but should be looked at separately and defined. What environment, client, workload and so on are you running?
Comment by gaurang jhawar [ 25/Oct/13 ]
My env is JDK 1.7.0_09
Client is my local machine ...

Couchbase server installed on aws ..

Reconnecting due to failure to connect to {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:423)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:261)
at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
2013-10-24 22:14:01.893 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1.
2013-10-24 22:14:05.903 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2013-10-24 22:14:09.632 INFO com.couchbase.client.ViewConnection: Added 172.31.9.124 to connect queue
Comment by Michael Nitschinger [ 25/Oct/13 ]
One thing to note here is that this is not a good idea. You have "the internet" between your computer (client) and the server which adds latency. Can you try running the application also in an AWS instance and see if the error goes away?
Comment by gaurang jhawar [ 25/Oct/13 ]
I gave the elastic amazon IP in the url and also tried what dragos tried but that also didn`t work for me .. I made ALL TCP ports on firewall ON ... And I am not sure why it reconnects to the IP or fails to connect and load data in the bucket.

But its the same case as dragos ... for production that`s not viable .. since not everything can be on AWS.

Also I can connect to the same instance via telnet and http(browser) .. I don`t know why I am facing this issue with Java 1.7.

"2013-10-24 22:14:01.893 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1.
2013-10-24 22:14:05.903 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
2013-10-24 22:14:09.632 INFO com.couchbase.client.ViewConnection: Added 172.31.9.124 to connect queue
2013-10-24 22:14:09.632 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
in INIT **************.
Here is the entity set : users
2013-10-24 22:14:09.868 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 0.
30 Seconds: Load is in progress
2013-10-24 22:14:09.993 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-24 22:14:12.037 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@6dae04e2
2013-10-24 22:14:12.037 INFO com.couchbase.client.CouchbaseConnection: Reconnecting due to failure to connect to {QA sa=172.31.9.124/172.31.9.124:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}
java.net.ConnectException: Connection timed out: no further information"

NOTE here it tried to reconnect to the client and says its inactive and I am not sure if this is a firewall issue or something else.
Comment by Michael Nitschinger [ 25/Oct/13 ]
Can you provide debug log information for the connect process? Maybe this helps pinning down where the issue is. (you can read here how to do it if you don't know already) :http://nitschinger.at/Logging-with-the-Couchbase-Java-Client

thanks
Comment by gaurang jhawar [ 25/Oct/13 ]
http://s000.tinyupload.com/index.php?file_id=46832814372422198765
Comment by Michael Nitschinger [ 25/Oct/13 ]
Thanks, can you also share the code you use?

In the code it looks like you are creating the CouchbaseClient object twice.
Please note that the bootstrapping succeeds (!) it is different from the ticket reported here.

Are you sure port 11210 is open from client to server? It looks like something is blocking it, http seems to work fine.
Comment by gaurang jhawar [ 25/Oct/13 ]
http://s000.tinyupload.com/index.php?file_id=34200463437173886967



ya the port is opened.

http://s000.tinyupload.com/index.php?file_id=06508245784961590671

Both inbound / outbound on aws as well as my local machine firewall ports are open
Comment by gaurang jhawar [ 26/Oct/13 ]
Any updateS?
Comment by couchbase-user [ 27/Oct/13 ]
This is a nightmare, can't believe no solution has been found yet for over a year now.....
Comment by Michael Nitschinger [ 28/Oct/13 ]
Hi couchbase-user, are you also running app/db over the internet or on a specific ec2 environment?
Comment by couchbase-user [ 28/Oct/13 ]
hey, actually I'm running them all on a local small network.
I am using ubuntu 12.4 on all machines and running java application on headnode to get view data and store them locally.
Comment by Michael Nitschinger [ 28/Oct/13 ]
Hm, I wonder why you are hitting this, I'm not sure its related.. do you get the same exception as the top of the ticket here? Can you provide debug logs so we can see at which point it is happening?

Maybe we need to tweak socket timeouts.
Comment by couchbase-user [ 28/Oct/13 ]
hey, here is my execution log:

28-Oct-2013 09:29:09 com.couchbase.client.CouchbaseProperties setPropertyFile
INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader.
2013-10-28 09:29:09.618 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.100:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:09.620 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.101:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:09.620 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=/192.168.0.102:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2013-10-28 09:29:14.635 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@7f09fd93
2013-10-28 09:29:14.635 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@79a5f739
2013-10-28 09:29:14.636 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@68e6ff0d
2013-10-28 09:29:14.656 INFO com.couchbase.client.ViewConnection: Added Cloud-assistant.local to connect queue
2013-10-28 09:29:19.663 INFO com.couchbase.client.ViewConnection: Added 192.168.0.102 to connect queue
2013-10-28 09:29:19.666 INFO com.couchbase.client.ViewConnection: Added 192.168.0.100 to connect queue
2013-10-28 09:29:19.667 INFO com.couchbase.client.CouchbaseClient: viewmode property isn't defined. Setting viewmode to production mode
2013-10-28 09:29:19.788 INFO com.couchbase.client.http.AsyncConnectionManager: Opening new Couchbase HTTP connection
2013-10-28 09:29:19.794 INFO com.couchbase.client.http.AsyncConnectionManager$ConnRequestCallback: /192.168.0.102:8092 - Session request successful
2013-10-28 09:29:24.815 ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out: [192.168.0.102/192.168.0.102:8092]
Exception in thread "main" java.lang.RuntimeException: Timed out waiting for operation
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:67)
    at com.couchbase.client.CouchbaseClient.getView(CouchbaseClient.java:483)
    at HelloWorld.Export_Couchbase_View(HelloWorld.java:85)
    at HelloWorld.main(HelloWorld.java:287)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
    at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:85)
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:74)
    at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:64)

I thought this is exactly the same as the error at the top of this thread....
Comment by Michael Nitschinger [ 28/Oct/13 ]
In your logs I can see that 192.168.0.101 is changed to Cloud-assistant.local. Could it be that this is a DNS issue in your environment?
Also please make sure that ports 8091, 8092 (!) and 11210 are reachable on each node.

Both the socket and connection timeout for views are set to 5 second which is already "quite high" if you want to run a real workload over it.
Comment by couchbase-user [ 28/Oct/13 ]
I'm not sure what the DNS has to do with it or how to check it, something to note here is that sometimes it works fine (very rare).
I have also just tried to change the switch but still the same problem.
I'm working on investigating the DNS and firewall issue. thanks
Comment by Michael Nitschinger [ 28/Oct/13 ]
Okay, if you go directly to the view url, how long does it take you to? like with curl... so we can see how far you are over the 5 seconds and if you are under it then there's something else going on.
Comment by couchbase-user [ 28/Oct/13 ]
I have added different servers names to hosts files in all servers and disabled the firewall on all of them and now it seems to work fine.
Thanks a lot for your help Michael.
Comment by Michael Nitschinger [ 28/Oct/13 ]
perfect, good to know it works now.
Comment by gaurang jhawar [ 28/Oct/13 ]
Any updates on my issue Michael Nitschinger. Thanks
Comment by Michael Nitschinger [ 29/Oct/13 ]
Since we fixed the suspected issue with "couchbase-user" in kind of the same way,
ca you first verify that

- All ports are reachable from all the boxes (11210, 8091, 8092)
- DNS settings are configured properly, this is especially important in EC2

If you look through the thread here, some of the people reporting this thing fixed once they made sure their environmental settings were okay. Can you double check?
Comment by gaurang jhawar [ 31/Oct/13 ]
Couchbase-user how did u configure hostname . can u tell me a step by step approach .. dont give me links to couchbase documentation that didn`t help ..

Michael .. the ports are open I am able to ping .. it connects via java and everything but the timeout is quite frequent ..



C:\Users\gaurang\Downloads>tcping 54.219.141.78 8091

Probing 54.219.141.78:8091/tcp - Port is open - time=21.692ms
Probing 54.219.141.78:8091/tcp - Port is open - time=19.357ms
Probing 54.219.141.78:8091/tcp - Port is open - time=15.414ms
Probing 54.219.141.78:8091/tcp - Port is open - time=16.225ms

Ping statistics for 54.219.141.78:8091
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milli-seconds:
     Minimum = 15.414ms, Maximum = 21.692ms, Average = 18.172ms

C:\Users\gaurang\Downloads>tcping 54.219.141.78 11210\

Probing 54.219.141.78:11210/tcp - Port is open - time=18.236ms
Probing 54.219.141.78:11210/tcp - Port is open - time=23.513ms
Probing 54.219.141.78:11210/tcp - Port is open - time=1218.881ms
Probing 54.219.141.78:11210/tcp - Port is open - time=14.712ms

Ping statistics for 54.219.141.78:11210
     4 probes sent.
     4 successful, 0 failed.
Approximate trip times in milli-seconds:
     Minimum = 14.712ms, Maximum = 1218.881ms, Average = 318.836ms

Comment by CathodTech [ 13/Dec/13 ]
We have been facing the same issue for couples of months - with Couchbase on RHEL, and we have just found out the solution that works for us.

We have 2 Tomcat nodes running; 1 with Java 1.7 and another with 1.6. Couchbase was deployed on another 4 servers in the same network. We simply defined all Couchbase nodes in /etc/hosts file in both Tomcat nodes as you normally would (with x.x.x.x hostname), then it magically works for us - without having to restart Tomcat, nor Couchbase nodes.
Comment by Michael Nitschinger [ 13/Dec/13 ]
Thanks for pointing this out, I'll see if I can get it into the official docs to aid other people. To me this really looks like java-related and network-related. Interestingly only a few people are hitting it.
Comment by CathodTech [ 14/Dec/13 ]
For the issue I faced, it looks more like Java API related. Perhaps, something to do with DNS (e.g., something like gethostbyname() / gethostbyaddr()), and how the API handles the case when IP has no hostname. Note that we only use IP address to add nodes into Couchbase cluster and Java on other servers also connect to Couchbase with IP address, not hostname.

From the trace, we found that java code has no problem connecting (via connection pool) with Couchbase nodes that has IP/hostname pair in /etc/hosts. But for Couchbase nodes that were not defined in /etc/hosts, it attempted to connect, then the connection dropped, and returned timeout issue.

My team also found that later when he restarted java to re-deploy, Couchbase connection attempts at start-up were much faster (on both Java 1.6 and 1.7).

For this issue, we didn't face it when deploying Java and Couchbase on the same server as, perhaps, it is common to have "127.0.0.1 localhost" defined in /etc/hosts.

We hope that the solution we found would help you and others as well. Not sure if others are facing the same scenario, though.
Comment by David Borsos [ 18/Dec/13 ]
I've ran into a very similar/the same situation with the latest couchbase (2.2.0) server and java client (1.2.3), I thought I share my experiences.

The problem I faced was when trying to call CouchbaseClient.getView(). This apparently goes to one node in the couchbase cluster and queries some view metadata over HTTP (roughly). However I got "ERROR com.couchbase.client.ViewNode$EventLogger: Connection timed out" and that pretty much killed my client there. Seeing the solutions here with the hosts file workaround, I tried that and it indeed worked, but it's not really a suitable solution for us, so I decided to see into this a little bit more. Here is what I found.

(Disclaimer: this worked for me, and although I found other ways around this they may not work for you, and I did these attempting to solve this particular issue and didn't really care about anything else these changes may break, so be careful!)

(0) Baseline, the environment specific thing that actually causes the issue is that call java api call InetAddress.getHostFromNameService(InetAddress addr, boolean check) returns very slowly if there's no hostname to be found for a given IP. I have no idea why is that in my environment, I suppose could be because of a bunch of different reasons. Slow here means that it takes 4-5 secs for me for this call to return with an unknown hostname

This comes into the picture when the couchbase client tries to get a view. It uses Apache Http client to do this, and the following things happen:
(1) When the request for the view gets created, eventually we get to BaseIOReactor.execute which instantiates SessionHandle, and registers the time when this happened (this will cause the timeout)
(2) Somewhere in the request processing, we actually DO hit getHostFromNameService. More specifically this happens because a RequestTargetHost interceptor runs and this injects the "Host" header into the HTTP request.
(3) some time after (2) we hit a timeout check which compares the timestamp recorded in (1) and the current time, obviously resulting in a timeout and the error in the client if (2) takes a long time as indicated

The ideal solution would obviously be to fix whatever is wrong with my DNS configs, but I'm not (yet) sure what's wrong (running Ubuntu 13.10 btw). And also make sure that our live environments won't have the same issue...

One workaround I found that might be generally usable is to override the JVM's DNS resolver, by starting the client with: -Dsun.net.spi.nameservice.provider.1=dns,sun This worked for me, but I don't know what side-effects it may have (generally this may not be a great idea). Additionally this resulted in a significant increase of startup speed as the couchbase client seems to do a lot of ip->hostname lookups when it's starting (are these really necessary?)

Also I'd like to ask from the Couchbase folks if it's really necessary to do this reverse DNS lookup for every HTTP request. I cloned your client from git, and removed the RequestTargetHost interceptor from these calls; that also solved the timeout problem, and all seems to work fine. I didn't do exhaustive testing on it, and this, too, can be environment specific - running my couchbase cluster in local VMs at the moment.

Thanks.
Comment by Michael Nitschinger [ 19/Dec/13 ]
Hi David,

thanks for the heads-up, that gives me a better idea of whats going on. I basically revamped the whole view connection management for the next 1.3 release which hopefully should address this to. Would you want to try it out based on a preliminary build?

The thing is that you can't remove the RequestTargetHost, but it only does the DNS lookup if HTTP.TARGET_HOST is not set, but we can infer that from our configuration. So the new code basically sets it before the request is put in the queue, once we've determined which node to ask, then it should be skipped.

I already have that locally, so if you want to try it out we can quickly see if it fixes the issue.
Comment by Michael Nitschinger [ 19/Dec/13 ]
The issue will be fixed with the overall changeset of jcbc-388, which will appear in - as it stands now - 1.3, slated for jan 2014.
Comment by David Borsos [ 19/Dec/13 ]
Michael,

That sounds great. I'd be happy to help you verify your new code; send it over or let me know how can I get it, I'll switch to it and see if it solves the problem. If we can get the new version in early January (I saw it's scheduled for the 7th) that's cool too, we won't be doing much during the holidays anyway :)

Thanks
Comment by Sergii [ 19/Dec/13 ]
Hi Michael,

i have run into the same issue with couchbase 2.2 on mac - Java 1.7

Can i have the changed jar to try the fix?

Thanks
Comment by Michael Nitschinger [ 31/Dec/13 ]
A fix for this has been merged into master and will be available in the next (1.3.0) release.
Comment by Michael Nitschinger [ 31/Dec/13 ]
The new code minimizes dns lookups to workaround the issue, but the environment still needs to be fixed properly :)
Comment by Oded Hassidi [ 21/Jul/14 ]
Was this fixed, since we are having issues with timeot on Java SDK v1.4.x, and we are using JDK 7?
This is not happen all the time, if I run a thread that runs many request it occur every 2-4 min
Comment by Michael Nitschinger [ 21/Jul/14 ]
This issue happened during startup, are you experiencing issues during startup? Where do you get the timeouts?
Comment by Oded Hassidi [ 21/Jul/14 ]
Thanks for the quick response.
The timeout are not at startup it happens during runtime. every 2-3 minutes I get the followed:

SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.RuntimeException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:75)
        at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:778)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for operation
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:93)
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:65)
at com.couchbase.client.internal.ViewFuture.get(ViewFuture.java:49)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72)
... 45 more




[JCBC-487] com.couchbase.client.protocol.views.Qery key construct Created: 07/Jul/14  Updated: 15/Jul/14  Resolved: 15/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 1.4.1, 1.4.3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: reinerRubin Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Hello.
There is a little bug in query.setKey. When

String login = "0987654321";
Query query = new Query();
query.setKey(login);

query will be with ?key=987654321 not 0987654321.

It's happen in query.quote()
(https://github.com/couchbase/couchbase-java-client/blob/1.4.3/src/main/java/com/couchbase/client/protocol/views/Query.java#L292)


Also in version 1.3.0 in similar situation (key = "0987654321") key will be key=0987654321 and in response
java.util.concurrent.ExecutionException: OperationException: SERVER: bad_request Reason: invalid UTF-8 JSON: {{error,garbage_after_value},\"0987654321\"}

// Excuse my bad English

 Comments   
Comment by Michael Nitschinger [ 07/Jul/14 ]
You should use ComplexKey instead of the string directly.

Try:

query.setKey(ComplexKey.of("0987654321");
Comment by reinerRubin [ 07/Jul/14 ]
>You should use ComplexKey instead of the string directly.
Thanks. It's work.
But I don't sure about setKey(String) behavior.
Comment by Michael Nitschinger [ 15/Jul/14 ]
Yeah, with the string its not so straightforward, thats why we added the complex key. for now please stick to it, it will be easier with 2.0
Comment by reinerRubin [ 15/Jul/14 ]
Ok. Thanks!




[JCBC-440] Inconsistent result of paginated query. Created: 26/Mar/14  Updated: 09/Jul/14

Status: Open
Project: Couchbase Java Client
Component/s: Core, Views
Affects Version/s: 1.3.2
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Major
Reporter: Vladislav Koroteev Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Red Hat Enterprise Linux Server release 6.3 (Santiago)
Java 7
Couchbase Server Community Edition 2.2.0


 Description   
I get a strange behavior with paginated query. While I paginate on the result set, I get duplicates of some documents. I'm doing the following in my code:

// force start indexing of view
query.setStale(Stale.FALSE);
query.setReduce(true);
ViewResponse respFromReduce = client.query(view, query);
String expectedCountFromScroll = null;
for (ViewRow viewRow : respFromReduce) {
    expectedCountFromScroll = viewRow.getValue();
}
query.setReduce(false);
query.setIncludeDocs(false);
//disable indexing of view from sdk
query.setStale(Stale.OK);
Paginator scroll = client.paginatedQuery(view, query, SCROLL_SIZE);
int docsFromScrollCnt = 0;
while (scroll.hasNext()) {
    ViewResponse response = scroll.next();
    docsFromScrollCnt += response.size();
}

And I see, that docsFromScrollCnt > expectedCountFromScroll.
I think, that while query Couchbase auto-indexing view, so I decide to disable auto-indexing by set viewUpdateDeamon's parameters. I set updateInterval to value, that greater then querying time, updateMinChanges and replicaUpdateMinChanges to zero.

It's not helped, but I noticed following interesting feature. Count of duplicate documents always equals n*SCROLL_SIZE.

How can I avoid duplicate of document from paginated query?

- See more at: http://www.couchbase.com/communities/q-and-a/strange-behavior-paginated-query#sthash.MLFf15Gx.dpuf

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
Hi,

I need further clarification on the use case to triage this.

- Are you running workload while doing the request? That is, are you updating documents? Keep in mind that even with stale=false, the indexer on the server updates the view in the background and the results can change during the pagination.

- What reduce function do you use?

I'm pretty sure this is because the data is updated in between the calls, remember that your reduce call is more or less "atomic", but if you paginate on results there is no cursor, you are getting the batches with potential indexer updates in between.
Comment by Vladislav Koroteev [ 08/Jul/14 ]
- Are you running workload while doing the request?
Yes. When I get document from result set I change field on which I make view. For example, I have map-function {emit(doc.a, null)} and I changed field "a" of document. But I disable auto-indexing by set viewUpdateDeamon's parameters with REST-API. So I think, that indexing don't started. Also I use stale=OK while paginating result set of view. So, I don't understand why I have such result of view.

- What reduce function do you use?
I use built-in _count function.

-I'm pretty sure this is because the data is updated in between the calls, remember that your reduce call is more or less "atomic"
Before my test, database is not workload. I see, that process of auto-indexing is finished, because I call from web-interface reduce function with stale=FALSE. After that I disable auto-indexing, that's why result of view confused me.
Comment by Michael Nitschinger [ 09/Jul/14 ]
Normally there is an indexer running in the background, how did you disable auto-indexing? Also, if you want stale data you need to use stale=OK (since stale=false will update the index). If you on the other hand need 100% accurate results you need to do stale=false AND PersistTo.MASTER on the write side.
Comment by Vladislav Koroteev [ 09/Jul/14 ]
-how did you disable auto-indexing?

From documentation http://docs.couchbase.com/couchbase-manual-2.2/#automated-index-updates.
I make such http-request "POST http://nodename:8091/settings/viewUpdateDaemon updateInterval=10000000&updateMinChanges=0"

I request view reduce function with stale=FALSE. So I make index is fresh. Then I request view with stale=OK in order to avoid indexing of view, but still get duplicates documents in result set. Another strange feature, that count of duplicates always equals N*SCROLL_SIZE. N is variable in different requests.




[JCBC-473] Flush support Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-471] Implement HELLO command Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-472] Implement support for snappy compression where supported Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-468] Implement EBNF-based N1QL DSL Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-469] Implement HTTP-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-470] Implement Carrier-based configuration management Created: 10/Jun/14  Updated: 07/Jul/14  Resolved: 07/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0-dp2
Security Level: Public

Type: New Feature Priority: Blocker
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-484] Publish docs for Java July 2014 release Created: 01/Jul/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Closed
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.4.3
Fix Version/s: 1.4.3
Security Level: Public

Type: Task Priority: Critical
Reporter: Amy Kurtzman Assignee: Amy Kurtzman
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 1h
Time Spent: Not Specified
Original Estimate: 1h


 Description   
Publish updated documentation, release notes, and javadocs.




[JCBC-482] Select proper replica node for getsFromReplica Created: 27/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-480] replica read with NOT_EXISTS never completes Created: 25/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-477] Pass view timeout down to get bulk when docs used Created: 23/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-476] Rework now misleading WARN message. Created: 23/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-475] Harden DefaultConfigurationProvider on restart and shutdown. Created: 17/Jun/14  Updated: 01/Jul/14  Resolved: 01/Jul/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3
Security Level: Public

Type: Improvement Priority: Major
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency




[JCBC-447] add feature test ensuring that E2BIG is returned on append above 20MB Created: 07/Apr/14  Updated: 23/Jun/14

Status: In Progress
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.3.2, 1.4.0-dp1
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Major
Reporter: Matt Ingenthron Assignee: Deepti Dawar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
depends on MB-10778 Append do not return the correct erro... Closed

 Description   
When continuing to append beyond the maximum value of 20MByte, we should verify that applications receive the correct error response.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
Do you want to take this on?
Comment by Deepti Dawar [ 23/Jun/14 ]
Up for review - http://review.couchbase.org/#/c/38675/1




[JCBC-467] Java client not aware of failed over node under certain circumstances Created: 10/Jun/14  Updated: 23/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: .next
Security Level: Public

Type: Task Priority: Critical
Reporter: Robert Waite Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File anotherMultithreadTest.java     Java Source File CouchbaseClientTesterFastAndThreaded.java     Java Source File CouchbaseClientTester.java     Java Source File CouchbaseClientTesterNewer.java     Text File logAfterSeemingFreeze.txt     Text File threadDumpDuringFreezeSunJVM.txt     Text File threadDumpDuringFreeze.txt    

 Description   
This is similar to http://www.couchbase.com/communities/q-and-a/java-client-not-aware-about...

This is using the 1.4.2 java client.

If I failover from the admin console while the node is still up... it does not reproduce. I found the best way to simulate the node suddenly becoming unresponsive is to use IP tables to block all traffic except SSH on port 22 like so:

To block a node:
iptables -A INPUT -p tcp --dport 22 -j ACCEPT; iptables -A INPUT -j DROP

To unblock a node:
iptables -F

It also seems to not reproduce if I remove the nested try/catch (i.e. don't try to read from the replica).

Failover seems to not be instantaneous... it takes 1-2 minutes with my hardware and setup. The following steps can be seen with the code below (might be reproducible with less steps but this seems consistent):

1) Create a two node cluster with 1 level of replication
2) Set the code below with the proper host names, bucket name and password
3) Run the code and you will see "Got From Master" 5 times
4) It will then pause and ask you to block traffic from the master node
5) Look at the admin console to see which node is the master for the key
6) Block the master node with iptables and then hit 'Enter'
7) Go back to the output and it will then output "Got From Replica" 10 times
8) It will then pause and ask for you to go to the admin console (on the replica node)
9) Wait until the master node is marked as "Down"
10) Once it is marked as "Down", fail over the master node
11) Go back to the console and hit 'Enter'

At this point the console should continue printing "Got From Replica". If you look at the admin console the replica node still has 0 items active and 1 item replicated. After 1-3 minutes it should suddenly say 1 item is active and 0 items are replicated (failover seems delayed). You will also notice at the same time that an exception started showing up in the console.

Expected: Once the node is fully failed over, it should no longer need to read from the replica and should read from the promoted master

Observed: It doesn't seem to be able to read from the master or the replica. It appears that the client is not marking the promoted replica as the new master.

Questions:

1) What is going on during the failover? I would have thought that failover would have been very fast and not take 1-5 minutes. Especially since I only have one item in the store
2) Anyone know of a workaround? If I catch the exception and rebuild the client... it works. But this would be horrible since the client is accessed by multiple threads.


 Comments   
Comment by Robert Waite [ 10/Jun/14 ]
Attached testing app. (CouchbaseClientTester.java)
Comment by Michael Nitschinger [ 10/Jun/14 ]
Thanks for the report, looking into it. Will provide an update as soon as I find something.
Comment by Michael Nitschinger [ 11/Jun/14 ]
Which server version are you using?
Comment by Michael Nitschinger [ 11/Jun/14 ]
If its 2.5.0 or 2.5.1, could you do me a favor and try setting "System.setProperty("cbclient.disableCarrierBootstrap", "true");" before the CouchbaseClient initialisation and report if it changes things? If its 2.2 nevermind.
Comment by Robert Waite [ 11/Jun/14 ]
It is 2.2.0. I can install 2.5.1 and give it a shot though. I had actually tried the disableCarrierBootstrap option but didn't realize that it wasn't available against the 2.2.0 server.
Comment by Robert Waite [ 11/Jun/14 ]
I upgraded the two nodes to 2.5.1 and added the disableCarrierBootstrap property before client creation. I was still able to repro the issue. There are some new details though:

1) The test will not always reproduce (even against the 2.2.0 server). Sometimes when you click failover from the admin console you will see the following message:

2014-06-11 10:01:35.889 INFO com.couchbase.client.CouchbaseConnectionFactory: Replacing current streaming node list [http://node1:8091/pools, http://node2:8091/pools] with [http://node1:8091/pools]
2014-06-11 10:01:35.894 INFO com.couchbase.client.CouchbaseConnection: Scheduling Node node3/10.1.1.10:11210for shutdown.

When this happens I normally see one more replica get, then it goes to the master and functions like I imagine it should. When testing against 2.5.1 this seems to happen 1 in 3 times. I think it was much less common with 2.2.0... maybe 1 in 8 times. My guess is that this could be related to how quickly failover seems to happen now in 2.5.1. Before... with 2.2.0 the pattern was wait about 1 minute to see the node marked as down, click failover and then see the active item show up on the other node after maybe 30 seconds at which point the error would show. With 2.5.1 the pattern is wait about 1 minute for the node to be marked as down, click failover and then immediately see the item show up on the other node as active and then the error will show.

2) Adding the disableCarrierBootstrap property did not seem to have any effect. I tried with and without maybe 5 times each. Both would show the error about 2 in 3 tests.

3) I altered the test to add the system property and to also continue trying replica gets continuously instead of pausing and waiting for the tester to click failover. Now steps 7+ from the original description are different. Once you complete 6 you should look at the admin console and wait for it to mark the node as down. Once it does, you click failover. If the error occurs then you will see exceptions. If the error does not happen you will see the INFO messages about replacing the current streaming node list and then it will start saying it is reading from the master instead of the replica (which is the desired behavior).
Comment by Robert Waite [ 11/Jun/14 ]
Attaching new version of testing app as CouchbaseClientTesterNewer.java
Comment by Michael Nitschinger [ 11/Jun/14 ]
Thanks very much for the additional info, I'll try to dig in this week and see what I can come up with. I'll let you know if I need anything else.

Again, very much appreciated!
Comment by Robert Waite [ 11/Jun/14 ]
Oh no! When I was testing the property change my POM was pointed at the 1.3.2 client instead of the 1.4.2 client. I have done three tests so far after adjusting the POM and now it seems to recover (I see the exception once but then it starts going to the master).

With this property change I see "Carrier config not available, bootstrapped through HTTP." in the log. Does this mean that the HTTP bootstrapping is not available from servers prior to 2.5.0?
Comment by Michael Nitschinger [ 11/Jun/14 ]
No thats good. It means that carrier config is not available prior to 2.5.0, so 2.2 will use HTTP. the 1.4 SDKs have a carrier -> http fallback, but with the sys property you could disable it on 2.5 as well.
Comment by Michael Nitschinger [ 11/Jun/14 ]
So with 1.4.2 it works?
Comment by Robert Waite [ 11/Jun/14 ]
1.4.2 client definitely failed against the 2.2.0 server. Maybe something about the faster failover in 2.5.1 changed what is happening? Let me reinstall 2.2.0 and confirm.
Comment by Robert Waite [ 11/Jun/14 ]
Confirmed that 1.4.2 client still fails against 2.2.0 server. When I click failover, instead of immediately seeing the data available on the promoted master... there is the 1-5 minute delay for the failover to actually complete. Then I see exceptions (not getting from replica or master).

However, with the disableCarrierBootstrap property set... after throwing the exception for 2-3 minutes it seems to recover. I am pretty sure it did not recover without that property set but I can validate that as well.

This likely is workable as a fix on my end. Do you know why failover is slow on 2.2.0 but seems to be so fast on 2.5.1?

Thanks again for your quick replies.
Comment by Robert Waite [ 11/Jun/14 ]
Hmm... even without the property set on 1.4.2 client against 2.2.0 server it seems to recover. I am wondering if this is related to queued up requests during the failover delaying or potentially blocking the client refreshing its node list.

For example... I get errors like this sometimes:

2014-06-11 11:48:53.284 WARN com.couchbase.client.CouchbaseConnection: Cancelling operation Cmd: 0 Opaque: 2682 Key: 27d528c1-0716-4f9d-98ef-ccfeaeee1b83because it has been retried (cloned) more than 100times.

I feel like the fullblown app I am testing with does not recover. Perhaps it is because it is more heavily threaded. I will try to work on a new testing app that 1) maybe reduces the timeout so that more failed requests can happen before failover is complete and maybe 2) try multiple threads all making requests against the client.

I am wondering if the problem is exacerbated in 2.2.0 because of the long failover and that perhaps there is a way to create it in 2.5.1. I am also wondering if there are cases that don't recover... because that is what originally made me dig into this.

I'll play with it some more and let you know.
Comment by Robert Waite [ 11/Jun/14 ]
Played with decreasing the timeout from the default of 2.5 seconds to 200 milliseconds. After maybe 3 or 4 runs it would always recover (after 1-3 minutes). I ran the gets with 6 threads and saw many new types of errors and it does not seem to recover (this is all against 2.2.0... I can try 2.5.1 a little later).

When I block traffic to the master node... it starts getting from replica... and then after a little while I see these sorts of errors:

Got From Replica
Got From Replica
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.937 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
Got From Replica
Got From Replica
Got From Replica
2014-06-11 12:50:04.942 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.
2014-06-11 12:50:04.942 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d.

At this point it is still getting data from replicas... but the warning starts showing up.

Once I kick off the failover, I see the exception start showing up (not able to get from replica or master) and finally some log messages show up about having trouble connecting happens and everything halts. The application is still running but nothing is happening... no errors. I will leave it for the next hour to see if it does somehow recover... but it is odd that timeouts no longer are spewing to the screen.

I will attach the threaded testing app and the log output when everything seems to halt.
Comment by Robert Waite [ 11/Jun/14 ]
Attaching quick timeout and threaded version of testing app. (CouchbaseClientTesterFastAndThreaded.java)
Comment by Robert Waite [ 11/Jun/14 ]
Attaching log file showing what leads up to the client seeming to lock up and no longer function. The last line in there is all that is on the screen but the java process is still alive. (logAfterSeemingFreeze.txt)
Comment by Michael Nitschinger [ 13/Jun/14 ]
Thanks for the detailed infos, I have a few more questions:

- Are you using views? Could be related to http://www.couchbase.com/issues/browse/MB-11146
- Can you do a thread dump after the observed lockup? maybe we can look in there and see whats going on.

Also, when you do a failover and all that stuff, you can check on the server logs and observe the timings where it says it started failover and once its done. Maybe this correlates with the info on the client side, I saw some other tickets (some internal support) where there were also longer failover pauses reported against 2.2 (from the server side). Maybe this helps us triage it further.

Just to clarify, against 2.5.X it seems to work fine and the observed behavior is only agains 2.2.?
Comment by Robert Waite [ 13/Jun/14 ]
We are not using views... just plain key/value pair lookups (although we are letting the client be responsible for serializing the objects... not serializing ourselves and passing strings).

I can do a thread dump.

The long failover pause only happened with 2.2.0... not 2.5.1. My guess was that it was related to MB-8039. Our installation of 2.2.0 is pretty vanilla (512MB per node limit on bucket... 1 level of replication... 2 nodes). I am pretty sure you will see the same behavior with the test code.

2.5.1 seemed to recover very quickly with property file change. My guess was that this was because the failover did not take 1-5 minutes to happen so less operations were outstanding client-side. This was just the test for recovery after failover. The issue where things locked up with the multi-threaded test was completely different... I have not tested the multi-threaded test against 2.5.1 yet.
Comment by Robert Waite [ 13/Jun/14 ]
Attached thread dump output.
Comment by Robert Waite [ 13/Jun/14 ]
Sorry... realized there were some actual hostnames in the threaddump... reattaching. (threadDumpDuringFreeze.txt)
Comment by Robert Waite [ 13/Jun/14 ]
Also attaching from Sun JVM in case there was a question about the other one being run in OpenJDK. (threadDumpDuringFreezeSunJVM.txt)
Comment by Robert Waite [ 13/Jun/14 ]
I switched the server back to 2.5.1 and in a number of tests... it did not seem to get into the frozen behavior in the client. Typically with the single thread test.. when I would click failover in 2.5.1 it would be less than half a second until the client seemed to pick up on the fact that there was a failover. With 2.2.0... it seemed to take 1-3 minutes (sometimes 5 minutes) to realize... but it would indeed failover.

It seems that the property being set had nothing to do with the fix... the only reason I thought it did was because of the issue where my POM was actually pointed to the 1.3.2 client instead of the 1.4.2 client.

So... 2.2.0 seems to have a number of issues with the various tests I have run... 2.5.1 seems resilient. It is possible that the issue is still in there though and is just harder to hit. I think the main reason 2.2.0 sees the issue so much more easily is related to the slow failover. Also... I think a unique feature of this test is that I am using replica gets and not just plain gets. I think at some point I tested commenting out the replica get and did not see the issue (but of course had no availability during that time).
Comment by Robert Waite [ 18/Jun/14 ]
Has anyone had a chance to run this? Even with 2.5.1 server and 1.4.2 client I see some odd behavior. When I cut the connection to the master node and see the replica gets... I will see a new error after maybe 20-40 seconds:

2014-06-18 11:57:28.711 WARN com.couchbase.client.CouchbaseConnection: sun.nio.ch.SelectionKeyImpl@5328f6ee exceeded continuous timeout threshold

Once this happens the data is no longer pulled from the replica and I get exceptions about timing out from any replica. This only happens when fully loaded with the 6 threads. If I back it down to 1 thread it doesn't seem to happen and didn't seem to happen at least a few times with 2 threads.

Might not be as high priority with the 2+ client coming out but I think similar tests should be run against it.
Comment by Robert Waite [ 18/Jun/14 ]
Also... I just recreated the freezing behavior in 2.5.1 server against the 1.4.2 client. I blocked a node with iptables and then just unblocked it without failing over. The client code hung. I have the thread dump but it looks pretty much the same as the others. Let me know if you want that specific dump.
Comment by Robert Waite [ 18/Jun/14 ]
I realize at this point that this ticket has ballooned to too many potential individual issues. It probably should have some subtasks created.

This is with 2.5.1 server against 1.4.2 client.

I created a new test that immediately starts performing gets with 8 threads. I then cut off a node with iptables. After about 20-40 seconds it will throw errors. If I trim it down to 1 thread I only see one error pop occaisionally, but in general it starts getting from the replica. If I bring the node back (without failover) it will start pulling from the master again after 20-60 seconds.

With this case I also added the disableCarrierBootstrap property back in and instead of reading from replicas after cutting the node off, it immediately throws errors. If I bring the node back it seems to stop processing.
Comment by Robert Waite [ 18/Jun/14 ]
Attaching newest test that was run against 2.5.1 server with 1.4.2 client. (anotherMultithreadTest.java)
Comment by Michael Nitschinger [ 23/Jun/14 ]
Thanks for your efforts on this, I tracked down some other issues and maybe this fixes yours as well. I'll look into it soon and report progress, hopefully we can nail that for 1.4.3.




[JCBC-424] investigate adding a ping or other check to idle connections Created: 02/Mar/14  Updated: 20/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
There have been a couple of reports of idle TCP connections causing operation failures. The underlying cause seems to be that some of these environments terminate seemingly idle TCP connections.

There are probably two solutions.
1) Don't let them go idle. On some existing thread or maybe with some kind of timer, do a version() operation or such regularly.
2) If it's possibly idle, send a ping before trying to send an op.

Of course, a 3rd solution is to automatically retry idempotent operations.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
I think we need to do this idle-ping style thing for another related bug edge case with CCCP, so I'll see if we can combine that for 1.4.2.
Comment by Mark Nunberg [ 02/Jun/14 ]
How about a simple ping() method? MySQL has it. We should have it in couchbase

btw ping won't help for those cccp edge cases either
Comment by Mark Nunberg [ 10/Jun/14 ]
I just realized this is attainable by something like the VERSION command :)
Comment by Michael Nitschinger [ 11/Jun/14 ]
Or even easier, what we do in 1.4.2 is a NOOP broadcast.




[JCBC-430] Situational Test Java CCCP upgrade : Results are showing auth issues causing latency with the java client. Created: 18/Mar/14  Updated: 19/Jun/14  Resolved: 19/Jun/14

Status: Closed
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deepti Dawar Assignee: Deepti Dawar
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency

 Description   
Description of specific scenarios is available in the respective SDKQE JIRA tasks.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
QE issues are closed, is this still an issue?
Comment by Deepti Dawar [ 04/Jun/14 ]
Will confirm after testing.
Comment by Deepti Dawar [ 19/Jun/14 ]
Latest test results are positive

https://docs.google.com/a/globallogic.com/spreadsheets/d/1IKxchnrsHQgBQZVF79y55zIk5MsNeuPnZ_iIZ6ptv8s/edit#gid=1382174636

Closing it.




[JCBC-445] 'Internal Server Error' in case of views for the latest jcbc Created: 02/Apr/14  Updated: 19/Jun/14  Resolved: 19/Jun/14

Status: Closed
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Critical
Reporter: Deepti Dawar Assignee: Deepti Dawar
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive 1.4-master_2.5.0.zip     Zip Archive 1.4-master_2.5.1.zip    
Issue Links:
Dependency
depends on MB-10743 View request may return 500 when node... Resolved

 Description   
Tested with the master on github i.e. 1.4-dp, reports have been shared.
Http return code 500 is being returned for the views because of which re-add and rebalance out scenarios are failing, each corresponding to the server version 2.5.0 and 2.5.1. Need to check with the server team for the 'Internal Server Error' that is being returned.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
Since the MB ticket was fixed, is this still an issue with 1.4.1 / the upcoming 1.4.2?
Comment by Deepti Dawar [ 04/Jun/14 ]
Will confirm after testing.
Comment by Deepti Dawar [ 19/Jun/14 ]
Latest tests are positive.

https://docs.google.com/a/globallogic.com/spreadsheets/d/1IKxchnrsHQgBQZVF79y55zIk5MsNeuPnZ_iIZ6ptv8s/edit#gid=1382174636

Hence, closing the issue.




[JCBC-449] Add an overview to the javadoc and the CouchbaseClient class Created: 15/Apr/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The HTML generated autodocs should have a better intro and get the user to the CouchbaseClient class more directly. For instance, if you go here:
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.0/index.html

One would have to click around for a bit to find what they're searching for.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
Are you sure this is possible with javadoc? I googled around a bit and didn't find anything.. well we could make it work with changing the files after generation or so.
Comment by Michael Nitschinger [ 17/Jun/14 ]
I fixed that in my build script, check out: http://www.couchbase.com/autodocs/couchbase-java-client-1.4.2/index.html




[JCBC-463] Java client occasionally leaks (non-daemon) netty IO threads after shutdown Created: 27/May/14  Updated: 17/Jun/14  Resolved: 17/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Task Priority: Major
Reporter: Dave Engberg Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
We're using CouchbaseClient v1.4 from a Java server with moderately high concurrency. When we shutdown the Couchbase client, it usually comes down cleanly, but we occasionally see lingering Netty threads
{noformat}
"New I/O worker #1" prio=10 tid=0x00000000014a3000 nid=0x58bd runnable [0x00007fc7c2f88000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059a590988> (a sun.nio.ch.Util$2) - locked <0x000000059a5909a0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a5af320> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:223) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) "Memcached IO over {MemcachedConnection to /10.6.70.3:11210 /10.6.70.2:11210 /10.6.70.1:11210}" prio=10 tid=0x00007fc7dcd48800 nid=0x58b8 runnable [0x00007fc7c38f1000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x000000059e331fe0> (a sun.nio.ch.Util$2) - locked <0x000000059e331ff8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000059a513d90> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:222) at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158)
{noformat}

This prevents clean Java server shutdown, and requires an aggressive "kill -9" to clean up. We figured out how to make the second thread start as 'daemon', but the "New I/O worker" thread can't be converted to daemon without gutting the client libs pretty heavily.

I haven't been able to reproduce this in isolated unit tests. We (Evernote) run around 500 Java7+Tomcat servers that receive a lot of activity over the course of a week, and see this problem come up in the wild around 20% of the time when we try to shut down. So it's a bit hard to narrow down the exact conditions that cause the thread leakage.

Our current workaround:
{noformat}
--- /tmp/BucketMonitor.java 2014-05-16 10:04:58.000000000 -0700
+++ src/main/java/com/couchbase/client/vbucket/BucketMonitor.java 2014-05-16 10:04:16.000000000 -0700
@@ -97,13 +97,28 @@
     this.configParser = configParser;
     this.host = cometStreamURI.getHost();
     this.port = cometStreamURI.getPort() == -1 ? 80 : cometStreamURI.getPort();
- factory = new NioClientSocketChannelFactory(Executors.newCachedThreadPool(),
- Executors.newCachedThreadPool());
+ factory = new NioClientSocketChannelFactory(newThreadPool(),
+ newThreadPool());
     this.headers = new HttpMessageHeaders();
       this.provider = provider;
   }
 
   /**
+ * Creates an executor based on a simple thread pool that only
+ * uses 'daemon' threads.
+ */
+ private java.util.concurrent.Executor newThreadPool() {
+ return Executors.newCachedThreadPool(
+ new java.util.concurrent.ThreadFactory() {
+ public Thread newThread(Runnable r) {
+ Thread t = new Thread(r);
+ t.setDaemon(true);
+ return t;
+ }
+ });
+ }
+
+ /**
    * Take any action required when the monitor appears to be disconnected.
{noformat}

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
I've got a similar report with the memcache IO thread, maybe this is related. I'll dig into it further and update the progress.
Comment by Michael Nitschinger [ 02/Jun/14 ]
I started work in both spy and jcbc to tackle this individually and harden the shutdown procedures in both.

http://review.couchbase.org/#/c/37726/
http://review.couchbase.org/#/c/37724/




[JCBC-474] Should be able to specify Transcoder to use when querying a view Created: 10/Jun/14  Updated: 16/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.1
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Minor
Reporter: raycardillo Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
All of the other methods in the interface support providing a Transcoder on each call and not necessarily relying on the default Transcoder configured in the connection builder. However, there is no way to do this when querying a view. This really only becomes an issue in combination with the "include documents" feature. When you're working with a document that has been Transcoded it will fail when you call getDocument() on the results unless you register a default Transcoder. It would be more consistent if we could pass a Transcoder to the getDocument() call (or configure it for the Query object)... and it would be even nicer if we could benefit from the duck typing logic to get the document as a JSON String if it's been identified by Couchbase as json.




[JCBC-466] Request full dataset when working with development views Created: 06/Jun/14  Updated: 10/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Core, Views
Affects Version/s: 1.4.1
Fix Version/s: .next
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Philipp Fehre Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Currently there is no way to request a full dataset when working with development views from the Java Client, this is solvable by just using the Production view and changing the path as well as appending the needed query parameter, but it would be nice to have the option for development views.

This came up and was requested at the Training in NY after Couchbase Live.




[JCBC-465] Unexpected behaviour on increment / decrement when combining with lock Created: 06/Jun/14  Updated: 10/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.4.1
Fix Version/s: .next
Security Level: Public

Type: Bug Priority: Trivial
Reporter: Philipp Fehre Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
When issuing an increment or decrement on a locked key (via getAndLock) the result is -1, which is according to documentation that the key hasn't been found. This is confusing when combining with default values. I created a gist for this to illustrate https://gist.github.com/sideshowcoder/8d3ae572bc8e0742ae64

I think a solution would be to add this to the documentation as expected.

This came up in the training in NY.




[JCBC-433] Java SDK Docs: comprehensive best practices example Created: 19/Mar/14  Updated: 02/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.2
Fix Version/s: .future
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Perry Krug Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
As a possible improvement to our existing docs, I was thinking that a comprehensive example of Java code showing the various components and strategies that an end-user would need to be aware of and how to handle those within our best practices would be helpful.

It's along the lines of the existing tutorial, but might actually be something separate since it's more about best practices rather than actually developing to Couchbase and can be used as a reference for seeing all those best practices come together.

We certainly have a lot of this existing already, I just haven't been able to see it all in one place to point a customer to.

From a format perspective, I think it would be helpful to have the full set of code on a page with various breakpoints/callouts/even inline comments that explain the various techniques and link further into the documentation for further reading. There can be commented-out sections for "other application logic <here>" as this really just needs to highlight the best practices of working with our Java library.

At least to include:
-Singleton creation and reuse of the client object across multiple threads. with error handling around if the connection can't be established
-Using set to overwrite some data and add() to enforce uniqueness, maybe even incr() and append() examples
-Using async operations with callbacks/listeners
-Using the "AtomicInteger" to track outstanding async operations before shutting down the object
-Proper exception catching and error handling around TMP_OOM, timeouts, failed add(), failed incr/decr, etc, possibly with some exponential back-off or other way of passing feedback up the stack
-Handling of "not found" from the get()
-Bulk gets
-Example of bulk set()s (different than a single async set, if the user knows that there are many items they want to set in bulk)
-Possibly putting some error handling code within the callbacks
-Turning on of our profiling feature
-Object->JSON encoding/decoding as well as handling of writing/reading non-JSON data to CB
-more?

Obviously there's a lot to put in here but I would also think that it doesn't need to be an all-or-nothing but can start with some simple pieces and be built upon. Certainly as we add more functionality or different ways of doing things this doc will have to be extended anyway.




[JCBC-461] Use alternative constructor to allow configuration of max New I/O client Worker thread count Created: 21/May/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Vanessa Ablaya Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Linux, Java 1.6


 Description   
We are running multiple instances of an application on a blade with 24 cores. Netty creates (2 * #PROCESSORS) New I/O Client Worker threads per couchbase client. This is causing high thread count, as each of our application instances does not have complete access to the resources 24 processors. I want to be able to configure the maximum number of threads Netty creates.

Line 100 of BucketMonitor.java uses the following constructor from the netty library:

 public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor)


I would like it to use the alternative constructor that netty provides, so that I can configure the workerCount value:

public NioClientSocketChannelFactory(
            Executor bossExecutor, Executor workerExecutor, int workerCount)



 Comments   
Comment by Michael Nitschinger [ 22/May/14 ]
That makes total sense. I'd rather not add an additional config setting for 1.* since the whole thing is revamaped for the 2.0 where the complete IO layer is based on netty and fully tunable.

What I could imagine though is a system property that you can set, which we could document. So people which run on those very powerful boxes can set it, and the others just don't have to care.
What do you think?
Comment by Michael Nitschinger [ 02/Jun/14 ]
Actually, I think we can trim that down since we are using it for streaming connections only. No need to provide that much. I'll investigate further and see what I can get into 1.4.2.
Comment by Michael Nitschinger [ 02/Jun/14 ]
Btw, two more things:

1) If you go to 1.4.1 and higher, it uses a different config fetching mechanism, so you should not see any of those threads created if you go with a server 2.5.0 and higher.
2) Even on your version, netty should create those threads only when needed, so do you mean you create lots of CouchbaseClient instances? You should reuse them as a singleton. Or do you see lots of those created even for one CouchbaseClient object? If so, this is a bug I think.

Can you provide more details on that? Maybe with a thread dump?
Comment by Michael Nitschinger [ 02/Jun/14 ]
http://review.couchbase.org/#/c/37722/
Comment by Michael Nitschinger [ 02/Jun/14 ]
We know only use one, so no need to configure. Please create a new ticket if you still then see issues with massive amount of threads generated, this is something else then for sure.




[JCBC-409] clarify behavior or *getFromReplica in javadoc Created: 03/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.0, 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Improvement Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I've seen questions a few times on how replica read behaves w.r.t. multiple replicas and recovery of the active location.

IIRC, the getFromReplica() will give you the first responder and the asyncGetFromReplica() will give you a future which has all the responses? The javadoc doesn't indicate this though, so it'd be great if we can clarify.




[JCBC-408] javadoc for ComplexKey class formatted incorrectly Created: 02/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.3.1
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Some of the code examples in the javadoc are not formatted correctly.

See:
http://www.couchbase.com/autodocs/couchbase-java-client-1.3.1/com/couchbase/client/protocol/views/ComplexKey.html




[JCBC-359] not all methods are documented on CouchbaseConnectionFactoryBuilder Created: 03/Sep/13  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Documentation
Affects Version/s: 1.1.9
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Just happened to notice several methods aren't documented or don't have docs punching through from the superclass.

http://www.couchbase.com/autodocs/couchbase-java-client-1.1.9/com/couchbase/client/CouchbaseConnectionFactoryBuilder.html




[JCBC-413] Race condition in cluster manager class Created: 11/Feb/14  Updated: 02/Jun/14  Resolved: 02/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Bug Priority: Critical
Reporter: Juan Manuel Musacchio Assignee: Michael Nitschinger
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I've been experiencing some connection issues ("Unable to connecto to cluster") while listing the couchbase buckets. After some debugging I realized that could be a race condition in the ClusterManager sendRequest method. The following are the involved lines (took it from master branch):

latch.countDown(); //line 474
success.set(true); //line 475
response.set(result); //line 476

The thing is that the mutex is decreased before setting sucess and response values, so later the code will check the success variable and could fail if success is not set before the thread starts (mutex is 0). I think the code should be:

success.set(true);
response.set(result);
latch.countDown();

 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
Thanks, good catch. I'll get the fix in 1.4.2
Comment by Michael Nitschinger [ 02/Jun/14 ]
http://review.couchbase.org/#/c/37719/1
Comment by Michael Nitschinger [ 02/Jun/14 ]
merged into master




[JCBC-401] Durability .get() with a custom Timeout higher than the default is ignored Created: 14/Jan/14  Updated: 02/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.3.0, 1.3.1
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Critical
Reporter: Michael Nitschinger Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Michael Nitschinger [ 02/Jun/14 ]
This is not trivial to fix and there is a workaround by setting a higher default timeout and using then custom timeouts on the async variant. It's not pretty but it works and will be overhauled in 2.0.




[JCBC-404] Views created via with carriage returns are not executable via web admin Created: 23/Jan/14  Updated: 02/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Major
Reporter: Brad Wood Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to
relates to MB-11275 Views created over REST API with CRLF... Resolved

 Description   
When looking at a view created via the Java SDK in the web admin, the "Show Results" button is greyed out when the map or reduce function contain a carriage return. The view can be executed via the SDK as well as the REST interface, but the "Show Results" button is disabled.

This was reported in the forums over a year ago:
http://www.couchbase.com/forums/thread/design-docs-views-created-rest-api-dont-work-show-results-grayed-out-404-error-client

I can't find a ticket for it and it doesn't appear to be resolved.

As a work around, I am manually replacing carriage returns (CFML code)
{code}
/**
* Normalize view functions to not have any carridge returns.
* The presence of ASCII code 13 in map or reduce functions will prevent you from
* being able to execute them from the Couchbase web admin.
* @viewFunction.hint The function string to normalize
*/
string function normalizeViewFunction( required string viewFunction ) {
var CR = chr(13);
var LF = chr(10);
var CRLF = CR & LF;
var CRLFPlaceholder = '__CRLF__';

// I'm doing this in two steps since if I just remove CR's that have no LF, that would completely remove line breaks and cause syntax issues
// but if I just replaced all CR's with LF's that would double all line breaks. Therefore, all CR's and CRLF's will be converted to an LF.
// Standalone LF's won't be touched.

// Set CRLF's aside for a moment
arguments.viewFunction = replace(arguments.viewFunction,CRLF,CRLFPlaceholder,'all');
// Replace single CR's with LF's
arguments.viewFunction = replace(arguments.viewFunction,CR,LF,'all');
// Put the CRLF's back as just LF's
arguments.viewFunction = replace(arguments.viewFunction,CRLFPlaceholder,LF,'all');

return arguments.viewFunction;
}
{code}


 Comments   
Comment by Brad Wood [ 23/Jan/14 ]
Ugh, you don't have Wiki markup enabled in JIRA. Please do so, it makes things so much prettier. I can't edit my ticket either to remove the unused "code" markup.
Comment by Michael Nitschinger [ 23/Jan/14 ]
Hi Brad,

do you mean that when you create view through the java SDK which has a CR in its name (design doc, view name?) it can't be used from the web UI, but otherwise works fine?

Either this is not allowed then we should check it or it is a bug on the server side.
Comment by Brad Wood [ 23/Jan/14 ]
No, the carriage returns are not in the name of the view or the design document-- they are in the map or reduce functions.

For instance, I might create a view with the CFML SDK (which proxies to the Java SDK) with the following code:

couchbase.saveView(
'beer',
'listBeersByBrewery',
'function (doc, meta) {
if ( doc.type == ''beer'' ) {
emit(doc.brewery_id, null);
}
}',
'_count'
);

Since I am on Windows, each line ending in the map function is a carriage return followed by a line feed. The view will create and will be executable via the SDK and via the REST interface, but not via the web admin. After I modified the CFML SDK to strip out ASCII code 13 from the map and reduce functions, they work as expected.
Comment by Michael Nitschinger [ 02/Jun/14 ]
While I think we could implement that, it could also be a bug on the server side. Let me open a issue there and the we'll figure it out.
Comment by Michael Nitschinger [ 02/Jun/14 ]
I've passed this on to the UI folks so they can look into. Depending on what comes out in the MB ticket (http://www.couchbase.com/issues/browse/MB-11275) we'll see if there is action needed on the java side.




[JCBC-405] Allow custom server ports in the test suite (was: apache http library throws 'badurl' error while running Java SDK ant test against cluster on non-default port.) Created: 23/Jan/14  Updated: 02/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 2.0
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Venu Uppalapati Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: CentOS 6.4 64-bit
RAM:256GB
Dual 2.9GHz 8-core Xeon E5-2690 for 32 total cores (16 + hyperthreading)


 Description   
This is part of multi-instance testing for customer 'A'.

Steps to reproduce:
1)Setup a 3 instances per physical machine. configure a cluster of 5 such instances. The instances are modified to run on non-default ports by modifying static config file. for example:
{rest_port,9000}.
{mccouch_port,9001}.
{memcached_port,12000}.
{memcached_dedicated_port,12001}.
{moxi_port,12002}.
{short_name,"ns11"}.
{ssl_rest_port,11000}.
{ssl_capi_port,11001}.
{ssl_proxy_downstream_port,11002}.
{ssl_proxy_upstream_port,11003}.

2)clone java SDK project. change the server ip address and server port number(internal memcached port) in the build.xml file to reflect the server address and non-default internal memcached port (12001 in this case).
example:sysproperty key="server.address_v6" value="${server.address_v6}"/>
            <sysproperty key="server.port_number" value="12001"/>

3) the junit tests in the Java SDK contain hardcoded REST port, so modify all of them to non-default REST port:
find . -name "*.java" -print0 | xargs -0 sed -i '' -e 's/8091/9000/g'

4)Run 'ant test'. the ClusterManager tests fail with following apache http library error. the library methods are called from ClusterManager.java

2014-01-23 16:35:03.786 WARN com.couchbase.client.ClusterManager: Cluster Response failed with:
    [junit] java.net.UnknownHostException: badurl
    [junit] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.validateAddress(DefaultConnectingIOReactor.java:245)
    [junit] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:265)
    [junit] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:141)
    [junit] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
    [junit] at com.couchbase.client.ClusterManager$2.run(ClusterManager.java:589)
    [junit] at java.lang.Thread.run(Thread.java:695)


 Comments   
Comment by Michael Nitschinger [ 23/Jan/14 ]
Alright, this will need some larger changes in the test suite, we didn't anticipate this back in the days.

Which ports do you need to have changed?
Comment by Venu Uppalapati [ 23/Jan/14 ]
All of the ports listed in step 1 of description need to be changed per instance.




[JCBC-435] update README info on maven repos Created: 21/Mar/14  Updated: 01/Jun/14  Resolved: 01/Jun/14

Status: Resolved
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.2
Security Level: Public

Type: Task Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I happened to look at the README earlier and it still has old maven repository info.

 Comments   
Comment by Michael Nitschinger [ 01/Jun/14 ]
The github master has now the 2.0 info and we have a link for 1.4 to the communities site.




[JCBC-416] Client throws exception "Cannot cache data larger than 20971520 bytes" when receiving uncompressed document from server Created: 17/Feb/14  Updated: 01/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: .future
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Steps to reproduce:
1) generate a 30MB document. compress the document, in my case the compressed size is ~15MB.
2) using a 3.0 client, send this compressed doc with datatype 0x02 to the server. server stores this as is.
3) using a 2.x client, request a GET for this document. The server uncompresses the doc since the request if from 2.x client that is not flex metadata aware and sends the raw doc to client.
4) client throws following exception and connection resets..java.lang.IllegalArgumentException: Cannot cache data larger than 20971520 bytes (you tried to cache a 31211520 byte object)
5) we should release appropriate hot patch or workaround information on how to handle this scenario from 2.x clients.




[JCBC-429] Possible race condition with add() multiple times on the same key Created: 14/Mar/14  Updated: 01/Jun/14  Resolved: 01/Jun/14

Status: Closed
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.3.2
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Michael Nitschinger Assignee: Unassigned
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt: start-finish

 Description   
public class CouchbaseAddTest {

    private ExecutorService pool;

    public static void main(String[] args) {
        new CouchbaseAddTest().run();
    }
    
    public void run() {
        // create a pool of 4 threads
        pool = Executors.newFixedThreadPool(4);
        
        // create 4 writers to run in parallel
        List<Future> fs = new ArrayList<Future>(4);
        for (int i=0; i<4; i++) {
                fs.add(pool.submit(new Writer(i+1)));
        }
        
        for (Future f : fs) {
            try {
                f.get();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        pool.shutdown();
    }
    
    private class Writer implements Runnable {
        private int id;
        private Gson gson;

        public Writer(int id) {
            this.id = id;
            gson = new Gson();
        }
        
        @Override
        public void run() {
            try {
                // Connect to the Cluster
                CouchbaseClient client = getClient();
                
                for (int i=0; i<1; i++) {
                    addDoc(client, i);

                    try {
                        Thread.sleep(200);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
               
                // Shutting down client
                client.shutdown();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        
        private CouchbaseClient getClient() throws IOException, URISyntaxException {
            
            CouchbaseConnectionFactoryBuilder builder = new CouchbaseConnectionFactoryBuilder();
            builder.setReadBufferSize(16384); //16384
            builder.setOpTimeout(60000); //60000
            builder.setFailureMode(FailureMode.Redistribute);
            builder. setOpQueueMaxBlockTime(5000); // add extra
            
            List<URI> couchServers = Arrays.asList(
                    new URI("http://server1:8091/pools"),
                    new URI("http://server2:8091/pools"),
                    new URI("http://server3:8091/pools"),
                    new URI("http://server4:8091/pools")
            );

            CouchbaseConnectionFactory connectionFactory =
                    builder.buildCouchbaseConnection(couchServers,
                            "usage", "", "");

            return new com.couchbase.client.CouchbaseClient(connectionFactory);
        }
        
        private void addDoc(CouchbaseClient client, int n) {
            Doc doc = null;
            String value = null;

            String key = String.format("dc1-daily-%d", n);
            if (get(client, key)==null) {
                doc = new Doc();
                value = gson.toJson(doc);
                add(client, key, value);
            }
            
            key = String.format("daily-%d", n);
            if (get(client, key)==null) {
                doc = new Doc();
                value = gson.toJson(doc);
                add(client, key, value);
            }
        }
        
        private Object get(CouchbaseClient client, String key) {
            return client.gets(key);
        }
        
        private void add(CouchbaseClient client, String key, String value) {
            
            OperationFuture f = client.add(key, value);
            OperationStatus status = null;
            try {
                status = f.getStatus();
            } catch (Exception e) {
                e.printStackTrace();
            }
            if (status!=null && status.isSuccess()) {
                System.out.println(String.format("%d Writer %d - Added %s", System.currentTimeMillis(), id, key));
            } else {
                System.out.println(String.format("%d Writer %d - Add failed %s", System.currentTimeMillis(), id, key));
            }

        }
    }
    
    private static class Doc {
        
        public String[] s = {
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890",
                "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890"
        };
    }
}



In some environments, this leads to duplicate adds being reported:

1393524003358 Writer 3 - Added dc1-daily-0
1393524003360 Writer 4 - Added dc1-daily-0
1393524003363 Writer 4 - Added daily-0
1393524003363 Writer 3 - Added daily-0
1393524003367 Writer 2 - Added dc1-daily-0
1393524003368 Writer 1 - Add failed dc1-daily-0
1393524003370 Writer 2 - Added daily-0
  

instead of the correct

1393524753507 Writer 1 - Added dc1-daily-0
1393524753507 Writer 2 - Add failed dc1-daily-0
1393524753507 Writer 4 - Add failed dc1-daily-0
1393524753507 Writer 3 - Add failed dc1-daily-0
1393524753510 Writer 2 - Added daily-0
1393524753510 Writer 4 - Add failed daily-0
1393524753510 Writer 1 - Add failed daily-0
1393524753511 Writer 3 - Add failed daily-0


reproduction was not yet successful

 Comments   
Comment by Matt Ingenthron [ 17/Mar/14 ]
I also tried to reproduce this, to no avail. One suspicion (originally raised by Michael) is perhaps the nodes being used for bootstrap aren't clustered or there is something unhealthy in the cluster?

It would be great to get a debug level log which will probably point to the issue. See http://docs.couchbase.com/couchbase-sdk-java-1.3/#configuring-logging for information on setting up logging.




[JCBC-402] Do you plan to add the java bindings for incr multi ? Created: 15/Jan/14  Updated: 01/Jun/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: