[JCBC-530] Exception when using java client both in Spark and MapReduce Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sun Chen Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: Hadoop,Dependencies,HttpCore, Spark,
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h
Environment: Haddop2.4.1,Spark0.13,Java1.7,Ubuntu12.04


 Description   
I am using Couchbase java client(version 1.4.4),I writed some simple mapreaduce job and spark job to write and read data from Couchbase.But I always got an exception.Here is the exception information

-------------------------------------------------------------------------------------------------------------------------------------------------
2014-08-25 13:12:56,048 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V
at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156)
at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:31)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:20)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
-------------------------------------------------------------------------------------------------------------------------------------------

14/08/09 10:19:32 WARN Utils: Your hostname, precise64 resolves to a loopback address: 127.0.1.1; using 192.168.33.10 instead (on interface eth1) 14/08/09 10:19:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/08/09 10:19:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-09 10:19:40.569 INFO net.spy.memcached.auth.AuthThread: Authenticated to /127.0.0.1:11210 2014-08-09 10:19:40.677 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication. 2014-08-09 10:19:40.693 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=localhost/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue 2014-08-09 10:19:40.702 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{bucket='Result', nodes=[http://127.0.0.1:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500} Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156) at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194) at com.eyespage.dc.etl.c2s.Run$.main(main.scala:16) at com.eyespage.dc.etl.c2s.Run.main(main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2014-08-09 10:19:40.836 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210 - See more at: http://www.couchbase.com/communities/q-and-a/exception-when-using-scala-call-java-sdk%EF%BC%8Cand-run-spark#sthash.CmL5dcRz.dpuf








[JCBC-519] Timeout Not Honored on First Call to Java asyncGetBulk Created: 11/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File App.java    
Issue Links:
Duplicate

 Description   
This was observed on Couchbase Server 2.2.0. The same behavior is not evident on Couchbase Server 2.5.1. It appears to be independent of any Java client version. The very first call of asyncGetBulk does not respect the timeout value, and continues to process and wait. Subsequent calls do properly respect the timeout value. One interesting note is that this only occurs when the timeout value is "close" to the actual execution time. That is, a very short timeout is indeed honored.

Steps to reproduce:

* Download a Java client, such as http://packages.couchbase.com/clients/java/1.4.3/Couchbase-Java-Client-1.4.3.zip

* Unzip

* Copy the attached App.java into the unzipped directory

* Compile and run:

javac -cp couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar App.java

java -cp .:couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar:jettison-1.1.jar:netty-3.5.5.Final.jar:commons-codec-1.5.jar:httpcore-4.3.jar:httpcore-nio-4.3.jar App > 143.txt

* Note the following output in the first group. It would be expected that the first call would timeout, but does not:

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=108ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=16ms


Exception:********************
Timeout future is: true
Operation timed out. - failing nodes: 10.4.2.107/10.4.2.107:11210, cbhost.domain/10.4.2.57:11210
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms


ASK - Determine why the first call does not honor the timeout, and ideally suggest a workaround for Couchbase Server 2.2.0, short of upgrading to Couchbase Server 2.5.1 (where the issue does not appear exist, and therefore looks to be a server-side issue)




 Comments   
Comment by Aleksey Kondratenko [ 11/Aug/14 ]
Not honoring timeouts cannot be server side issue. Passing to Michael who owns couchbase java sdk
Comment by Jeff Dillon [ 12/Aug/14 ]
Please disregard my previous comments with respect to being apparently fixed with Couchbase Server 2.5.1. This behavior evidences itself across all CB server versions, and also reproduces with CB Java client 1.4.4. In my latest 2.5.1 test, I set the timeout to 30 ms, and see the same incorrect behavior. So this is indeed a client issue (as noted), apologies on the initial misinformation.
Comment by Michael Nitschinger [ 14/Aug/14 ]
A few remarks:

- The simple benchmark also profiles logging and so forth.
- System.currentTimeMillis() precision is sometimes beyond all joke, always use nanotime.

So something like this is more accurate:

        while (low == 10) {
            try {
                startTime = System.nanoTime();
                future = client.asyncGetBulk(Main.getKeys());
                future.get(timeout, TimeUnit.MILLISECONDS);
                endTime = System.nanoTime();
                System.out.println("Timeout future is: " + future.isTimeout());
                System.out.println("Fetched " + Main.getKeys().size() + " keys");
            } catch (Exception e) {
                System.out.println("\nException:********************\nTimeout future is: " + future.isTimeout());
                System.out.println(e.getMessage());
            }
            operationTime = endTime - startTime;
            System.out.println("Timeout set to:" + timeout + "\ngetMulti : Time taken=" + TimeUnit.NANOSECONDS.toMillis(operationTime) + "ms\n");
        }

Now also note that profiling on the blocking .get() call not only waits on the latch where the timeout is applied, but it also puts all data together and so forth. So as far as I can see it the timeout is honored, its just not what you benchmark that close. Keep in mind that the first few calls are subject t JVM warmup, optimizations and so forth which can take some time. It's not a huge surprise that the first few calls are slightly slower, but I guess thats not what the JVM optimizes for compared to a long-running server.

Does that help?
Comment by Jeff Dillon [ 14/Aug/14 ]
I am seeing the same behavior when using nanoTime. And it's not slightly slower, it's an order of magnitude difference. And I don't think it's JVM warmup either, as different timeout values show consistent differences every time. Using a short time out value works as expected, but is not the timeout value they desire. When the timeout is "close" to the actual execution time, it experiences this behavior. This is what I'm seeing in my tests, and is what the customer is seeing also.
Comment by Matt Ingenthron [ 20/Aug/14 ]
Jeff Dillon: I understand from James this has been observed on 2.5.1 now as well. Can you confirm and update it appropriately? Also, I've moved it to JCBC, since the investigation has lead here.
Comment by Jeff Dillon [ 20/Aug/14 ]
Yes, please see my previous comment from 12/Aug/14 1:47 PM




[JCBC-520] Running SSL Test on 2.0 Client hangs when certificate is not valid Created: 20/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The SSL test I make can be seen from
https://github.com/weilliu/couchbase-java-client/commit/97c3144692627d6e657a0cc157f5f9aa644de0f2
http://review.couchbase.org/#/c/40763/

Currently if we enable ssl, and when it hits a certificate which is not valid/not exist, the test would hang


11:38:16.868 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STARTED
11:38:17.843 [DEBUG] [TestEventLogger]
11:38:17.843 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STANDARD_OUT
11:38:17.843 [DEBUG] [TestEventLogger] File copied!
11:38:17.844 [DEBUG] [TestEventLogger] Reconnect with the Server
> 11:39:06.2 [ DEB] [org.gradle.process.internal.ExecHandleRunner] Abort requested. Destroying process: Gradle Worker 1.
> Building 85% > :integrationTest > 29 tests completed, 29 skippedWei-Lis-MacBook-Pro:couchbase-java-client wei-li$






[JCBC-543] IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest Created: 05/Sep/14  Updated: 05/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core, Infrastructure
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Morris Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This could be user error, but when trying the couchbae hellworld tutorial: http://docs.couchbase.com/prebuilt/java-sdk-2.0-beta/topics/hello-couchbase.html, I receive the following exception:

Sep 05, 2014 10:30:02 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Connected to Node 127.0.0.1
Sep 05, 2014 10:30:07 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Disconnected from Node 127.0.0.1
Sep 05, 2014 10:30:07 AM com.couchbase.client.core.node.CouchbaseNode$5 call
INFO: Connected to Node jmorris-pc
Sep 05, 2014 10:30:07 AM com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler handleEventException
SEVERE: Exception processing: 2 com.couchbase.client.core.RequestEvent@7bcfceb7
java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.core.node.locate.BinaryLocator.locateForCouchbaseBucket(BinaryLocator.java:126)
at com.couchbase.client.core.node.locate.BinaryLocator.locate(BinaryLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

Exception in thread "pool-1-thread-2" java.lang.RuntimeException: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.deps.com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IllegalStateException: Node not found for requestcom.couchbase.client.core.message.binary.UpsertRequest@9c94115
at com.couchbase.client.core.node.locate.BinaryLocator.locateForCouchbaseBucket(BinaryLocator.java:126)
at com.couchbase.client.core.node.locate.BinaryLocator.locate(BinaryLocator.java:63)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:148)
at com.couchbase.client.core.RequestHandler.onEvent(RequestHandler.java:68)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
... 3 more

Additional logging included. I am connecting to localhost CB server 2.5.




[JCBC-541] incr() returns -1 Created: 02/Sep/14  Updated: 09/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.2
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: RICHARDS PETER Assignee: Matt Ingenthron
Resolution: Unresolved Votes: 0
Labels: info-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Hi,

I am using incr() of couchbase client to increase a counter key in a bucket. I found this method very useful in some of our usecases. Around 150 threads from four machines increments current value of counter by 10 and use the values for some internal logic. Initially everything worked as expected. However a few days back I realized that couchbase returns -1 for some of the incr() method calls. That is how I noticed that incr() in MemcachedClient could return -1 if there is a failure to update the counter:
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.3/net/spy/memcached/MemcachedClient.html

I have a few queries about this return value:
1. Could you please tell me the cases in which incr() would return -1? I identified one case. If the RAM quota allocated to the bucket is exhausted, we are likely to get -1. During this time addition of key-value pairs into the same bucket using add() will return false through the OperationFuture object. What are the other cases that would return -1 in the incr()?

2. Could you please tell me how the following method call works?
http://www.couchbase.com/autodocs/couchbase-java-client-1.4.3/net/spy/memcached/MemcachedClient.html#incr%28java.lang.String,%20long,%20long,%20int%29

I would like to know when the default value would be returned and when -1 would be returned. I think the default value would be returned if it is possible to insert the value into couchbase and -1 would be returned if it is not possible to retrieve the counter and update it. Am I correct?

3. Till now my project did not have a use case to increment negative values. However if somebody wants to increment -2 by 1 then the result would be -1. How will the client call be able to distinguish failure and actual increment in this case? Wouldn't it be better to throw a checked exception from incr() than returning -1? I think with checked exception we can provide more information about the cause of failure.

Richards Peter.




[JCBC-548] user agent string is incorrect and doesn't work with some older HTTP core Created: 10/Sep/14  Updated: 10/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Matt Ingenthron Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
I found that the current client can't be used in CDH5 because a JavaLangNoSuchMethodError will be thrown with org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V since the older 4.2 Apache httpcore does not have that method of defining a string and it takes precedence in the classpath.

In the process, I found that this header wasn't being set correctly anyway.

Filing this issue to remove the header. If we get a chance, it should be fixed by re-adding with reflection or whatever method 4.2 provides.




[JCBC-547] Paginator hasNext() works incorrectly on small results Created: 09/Sep/14  Updated: 09/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Views
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: raycardillo Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: javaclient
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
While working with the Paginator in our QA system I traced down a bug that will only happen when the page size is greater than the number of items in the underlying view/query. We wanted to call hasNext() multiple times because one layer of code validates that it has a good Paginator (due to other problems we were having with our cluster). However, because the actual response is fetched during hasNext(), and because the FINISHED state is set as a result of that, only the first hasNext() will return true even though we have not retrieved the response with next() yet.

For example, we should be able to call, hasNext() multiple times, and then state is only changed to indicate FINISHED after we actually retrieve the response using next().

After tracing this bug down, I can see that hasNext() returns false right away if FINISHED is true, and I can see that the FINISHED state is set during the fetch that happens during hasNext(). So when there is less than one page of data available, the first hasNext() will return true, and the next hasNext() will return false, which is a problem if the consumer has not actually called next() to fetch the response.

I think the simple fix here is to only set the FINISHED state in next() so you know that the data has actually been retrieved by the consumer. That way the consumer is free to ask hasNext() as many times as required for their logic, and it will not effect internal state until next() is called.




[JCBC-550] Java 2.0 SDK Backpressure Exception Created: 11/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
This is a known issue and a fix is coming soon.
I work around the issue by waiting a bit longer between ops. It does reduces the number of backpressue exceptions.

Will run the test again after the fix is pushed.





[JCBC-552] Support other sources for key store Created: 12/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Cihan Biyikoglu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified





[JCBC-549] Implement setHashAlg() / Implement ability to change hashing algorithm Created: 11/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: New Feature Priority: Major
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Gantt: start-finish
Relates to
relates to PYCBC-250 support for distributing data by hashkey Open

 Description   
CouchbaseConnectionFactoryBuilder.setHashAlg() is currently not operational.




[JCBC-553] 2.0.0.beta hangs on 2nd query Created: 12/Sep/14  Updated: 12/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-beta
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Sean Colonello Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Client is Windows 8.1, server is CentOS 6 running 3.0 beta.

Attachments: Text File two-queries-hang.log    
Issue Links:
Duplicate
is duplicated by JVMCBC-28 ViewHandler can end up not completing... Resolved

 Description   
The following code hangs on the 2nd query.

CouchbaseCluster cluster = CouchbaseCluster.fromConnectionString("couchbase://192.168.1.75");
Bucket bucket = cluster.openBucket("trade-unittest", "blah").toBlocking().first();
bucket
.query(ViewQuery.from("devOnly", "allDocs"))
.flatMap(result -> result.rows())
.toBlocking()
.forEach(row -> logger.log(Level.INFO, row.id()));
bucket
.query(ViewQuery.from("devOnly", "allDocs"))
.flatMap(result -> result.rows())
.toBlocking()
.forEach(row -> logger.log(Level.INFO, row.id()));
cluster.disconnect();





[JCBC-502] Sometimes couchbase client stuck Created: 29/Jul/14  Updated: 15/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: Infrastructure
Affects Version/s: 2.0-dp2
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Oded Hassidi Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Mac OSX as client


 Description   
Sometimes I run the client like in the examples in the community blog and the client stuck and Doesn't return. To validate that I run the client several times one after the the other and sometimes it get stuck and some don't

 Comments   
Comment by Michael Nitschinger [ 11/Aug/14 ]
Can you provide logging with level set to FINEST? that would be of great help, thanks.
Comment by Oded Hassidi [ 12/Aug/14 ]
I am running my test in a main, and I have debug level entries. Is that enough?
Comment by Michael Nitschinger [ 12/Aug/14 ]
Yes send over whatever you have, and the code would also be good.
Comment by Oded Hassidi [ 12/Aug/14 ]
Code:


public class Main {
public static void main(String[] args) throws IOException {
      
        System.setProperty("com.couchbase.client.queryEnabled", "true");

        CouchbaseCluster cc = new CouchbaseCluster("obbe-centos3.tlv.lpnet.com", "obbe-centos1.tlv.lpnet.com", "obbe-centos2.tlv.lpnet.com");

        final Bucket[] bucket = new CouchbaseBucket[1];
        System.out.println("******** Opening the Bucket");
        cc.openBucket("visitors-odedh").subscribe(b -> {
            System.out.println("################### Bucket is opened: ");
                System.out.println(b.toString());
                bucket[0] = b;
        });


        System.out.println("enter something: ");
        System.in.read();
    }
}



log:

/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/bin/java -Didea.launcher.port=7534 "-Didea.launcher.bin.path=/Applications/IntelliJ IDEA 14 EAP.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dal/target/classes:/Users/odedh/.m2/repository/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar:/Users/odedh/.m2/repository/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar:/Users/odedh/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/Users/odedh/Projects/visitorfeed/branches/n1ql/dao/target/classes:/Users/odedh/.m2/repository/com/liveperson/framework/rest-api-data-objects/2.0.2.11/rest-api-data-objects-2.0.2.11.jar:/Users/odedh/.m2/repository/com/liveperson/RestApiCommon/1.0.2-6/RestApiCommon-1.0.2-6.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-core/1.9.1/jersey-core-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-client/1.9.1/jersey-client-1.9.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-server/1.9.1/jersey-server-1.9.1.jar:/Users/odedh/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/odedh/.m2/repository/com/sun/jersey/jersey-json/1.9.1/jersey-json-1.9.1.jar:/Users/odedh/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/odedh/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/odedh/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/Users/odedh/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/odedh/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/Users/odedh/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.3/jackson-core-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.3/jackson-mapper-asl-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/Users/odedh/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/Users/odedh/.m2/repository/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar:/Users/odedh/.m2/repository/org/springframework/spring-context/4.0.6.RELEASE/spring-context-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-aop/4.0.6.RELEASE/spring-aop-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/odedh/.m2/repository/org/springframework/spring-beans/4.0.6.RELEASE/spring-beans-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-core/4.0.6.RELEASE/spring-core-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/org/springframework/spring-expression/4.0.6.RELEASE/spring-expression-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/couchbase/client/couchbase-client/2.0.0-dp2/couchbase-client-2.0.0-dp2.jar:/Users/odedh/.m2/repository/com/couchbase/client/core-io/0.2/core-io-0.2.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.4.0/jackson-databind-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.0/jackson-annotations-2.4.0.jar:/Users/odedh/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.0/jackson-core-2.4.0.jar:/Users/odedh/.m2/repository/com/lmax/disruptor/3.2.1/disruptor-3.2.1.jar:/Users/odedh/.m2/repository/com/netflix/rxjava/rxjava-core/0.19.2/rxjava-core-0.19.2.jar:/Users/odedh/.m2/repository/com/typesafe/config/1.2.1/config-1.2.1.jar:/Users/odedh/.m2/repository/io/netty/netty-all/4.0.21.Final/netty-all-4.0.21.Final.jar:/Users/odedh/.m2/repository/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/Users/odedh/.m2/repository/org/springframework/spring-test/4.0.6.RELEASE/spring-test-4.0.6.RELEASE.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring/4.0.0.1/SnmpMonitoring-4.0.0.1.jar:/Users/odedh/.m2/repository/com/liveperson/SnmpMonitoring-api/4.0.0.1/SnmpMonitoring-api-4.0.0.1.jar:/Users/odedh/.m2/repository/org/apache/xmlbeans/xmlbeans/2.4.0/xmlbeans-2.4.0.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus/3.0.0.2/ServiceStatus-3.0.0.2.jar:/Users/odedh/.m2/repository/com/liveperson/ServiceStatus-API/3.0.0.2/ServiceStatus-API-3.0.0.2.jar:/Applications/IntelliJ IDEA 14 EAP.app/Contents/lib/idea_rt.jar" com.intellij.rt.execution.application.AppMain com.liveperson.visitorfeed.dal.couchbase.CouchbaseVisitorDao
15:55:05.598 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
15:55:05.604 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
15:55:05.616 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
15:55:05.617 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: true
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - UID: 733553586
15:55:05.624 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
15:55:05.625 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: unavailable
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes. Please check the configuration for better performance.
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: /var/folders/5f/wk7ckrbx1f30521n8rtxndzwnvk8xk/T (java.io.tmpdir)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
15:55:05.626 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
15:55:05.640 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
15:55:05.712 [main] DEBUG c.c.c.c.config.ConfigurationProvider - Setting seed hosts to [obbe-centos1.tlv.lpnet.com/192.168.15.120, obbe-centos3.tlv.lpnet.com/192.168.15.190, obbe-centos2.tlv.lpnet.com/192.168.15.2]
******** Opening the Bucket
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 8
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
15:55:05.754 [RxComputationThreadPool-1] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
enter something:
15:55:05.780 [RxComputationThreadPool-1] DEBUG i.n.util.internal.ThreadLocalRandom - -Dio.netty.initialSeedUniquifier: 0x9729d41313197dd7
15:55:05.785 [RxComputationThreadPool-1] DEBUG i.n.channel.ChannelOutboundBuffer - -Dio.netty.threadLocalDirectBufferSize: 65536
15:55:05.786 [RxComputationThreadPool-1] DEBUG io.netty.util.Recycler - -Dio.netty.recycler.maxCapacity.default: 262144
15:55:05.795 [RxComputationThreadPool-1] DEBUG io.netty.buffer.ByteBufUtil - -Dio.netty.allocator.type: unpooled
15:55:05.880 [nioEventLoopGroup-2-2] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetectionLevel: simple
15:55:05.981 [nioEventLoopGroup-2-2] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-6] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos3.tlv.lpnet.com/192.168.15.190:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-3] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos1.tlv.lpnet.com/192.168.15.120:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.983 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.983 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.983 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.983 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:11210
15:55:05.984 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:11210
15:55:05.984 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.984 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Disconnected from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.984 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.984 [nioEventLoopGroup-2-8] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Disconnected (CONNECTING) from Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos1.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.c.c.e.binary.BinaryHelloClient - obbe-centos2.tlv.lpnet.com/192.168.15.2:11210 Hello not successful (Response Status: 129), falling back to no datatype.
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos3.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to BinaryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:11210
15:55:05.985 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (DEGRADED) to BinaryService obbe-centos2.tlv.lpnet.com
15:55:05.985 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:05.985 [nioEventLoopGroup-2-1] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.985 [nioEventLoopGroup-2-1] DEBUG com.couchbase.client.core.node.Node - Connected (DISCONNECTED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:05.994 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:05.996 [pool-1-thread-1] DEBUG c.c.c.c.config.loader.CarrierLoader - Successfully loaded config through carrier.
15:55:06.238 [RxComputationThreadPool-2] DEBUG c.c.c.c.config.ConfigurationProvider - Applying new configuration DefaultCouchbaseBucketConfig{name='visitors-odedh', locator=null, uri='/pools/default/buckets/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', streamingUri='/pools/default/bucketsStreaming/visitors-odedh?bucket_uuid=b496dc8a7cbae31046c914fff06dea2f', nodeInfo=[NodeInfo{viewUri='http://obbe-centos1.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos1.tlv.lpnet.com/192.168.15.120, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos2.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos2.tlv.lpnet.com/192.168.15.2, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}, NodeInfo{viewUri='http://obbe-centos3.tlv.lpnet.com:8092/visitors-odedh&#39;, hostname=obbe-centos3.tlv.lpnet.com/192.168.15.190, configPort=8091, directServices={VIEW=8092, CONFIG=8091, BINARY=11210}, sslServices={}}], partitionInfo=PartitionInfo{numberOfReplicas=1, partitionHosts=[obbe-centos1.tlv.lpnet.com, obbe-centos2.tlv.lpnet.com, obbe-centos3.tlv.lpnet.com], partitions=[[m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [1]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 0, r: [2]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [0]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 1, r: [2]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [0]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]], [m: 2, r: [1]]], forwardPartitions=[]}, tainted=false, rev=2070}
15:55:06.256 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8091
15:55:06.257 [nioEventLoopGroup-2-7] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos2.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8091
15:55:06.257 [nioEventLoopGroup-2-4] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos1.tlv.lpnet.com
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8092
15:55:06.257 [nioEventLoopGroup-2-6] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos2.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8092
15:55:06.258 [nioEventLoopGroup-2-1] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos3.tlv.lpnet.com
15:55:06.258 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ViewEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8092
15:55:06.259 [nioEventLoopGroup-2-3] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ViewService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos1.tlv.lpnet.com/192.168.15.120:8093
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos2.tlv.lpnet.com/192.168.15.2:8093
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos1.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos2.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.endpoint.Endpoint - Connected to QueryEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8093
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-8] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to QueryService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos1.tlv.lpnet.com/192.168.15.120
15:55:06.266 [nioEventLoopGroup-2-5] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos2.tlv.lpnet.com/192.168.15.2
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.endpoint.Endpoint - Connected to ConfigEndpoint obbe-centos3.tlv.lpnet.com/192.168.15.190:8091
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG c.c.client.core.service.Service - Connected (CONNECTING) to ConfigService obbe-centos3.tlv.lpnet.com
15:55:06.266 [nioEventLoopGroup-2-2] INFO com.couchbase.client.core.node.Node - Connected to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
15:55:06.266 [nioEventLoopGroup-2-2] DEBUG com.couchbase.client.core.node.Node - Connected (DEGRADED) to Node obbe-centos3.tlv.lpnet.com/192.168.15.190
Comment by Michael Nitschinger [ 21/Aug/14 ]
Hi,

can you please retry with dp3 and see if the issue still persists? If so, please also provide the debug logs - thanks! Also, can you specify what "hangs" means? Your println does never show?
Comment by Oded Hassidi [ 24/Aug/14 ]
Thanks, yes my println doesn't ever show!
Will check it with the beta, sorry for the delay I was OOO couple of days :)
Comment by Michael Nitschinger [ 04/Sep/14 ]
Hey, any update on this?
Comment by Michael Nitschinger [ 15/Sep/14 ]
I've removed the fix version since it's not clear if its a bug or not anymore. Please provide any details if you can as soon as possible so we can fix it if needed thanks!




[JCBC-546] Expected Behavior of FailureMode.Cancel Created: 08/Sep/14  Updated: 16/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Matt Ingenthron
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
Customer has questions on the behavior of FailureMode.Cancel:

1) Is TimeoutExceptionThreshold irrelevant when using FailureMode.Redistribute ? The node is not deemed down even when the number of consecutive failures exceed the threshold.
2) In what scenarios is it advisable to use Redistribute ?
3) Whats the .NET equivalent.




[JCBC-536] Interpretation and Performance Impact of Client Profiling Created: 29/Aug/14  Updated: 16/Sep/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Sergey Avseyev
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates to

 Description   
Customer is using client side profiling as suggested in:

http://blog.couchbase.com/whats-new-couchbase-java-sdk-12

Do we have documentation on the Meters, Counters and Histogram output? What would the performance impact be, if any? Can this be safely left in place in a Production environment, for example?

 Comments   
Comment by Jeff Dillon [ 02/Sep/14 ]
Here are specific asks:

1) We need documentation on counters, meters and histogram output.
2) Can we run clients with MetricType = PERFORMANCE for extended duration? What's the overhead?
3) Can we change the net.spy.metrics.reporter.type to hook it up to Graphite? The metrics-core library does offer support for a GraphiteReporter. Reference: http://metrics.dropwizard.io/manual/graphite/#manual-graphite
4) Is it possible to add custom metrics over and above what's being gathered by the Couchbase java client? If so, how?
Comment by Deepu Bhatia [ 09/Sep/14 ]
Hi Michael, Matt,
Can we please get an update on this?

Thanks,
-Deepu
Comment by Michael Nitschinger [ 10/Sep/14 ]
Hi,

sorry, I've been quite busy lately...

Currently there is not much more documentation available, and metrics can be used in production environments. That said, while the metrics library is designed to be used in production it is not "zero overhead" and should be used with care. PERFORMANCE just logs less information, only those needed to find performance infos.

wrt to graphite: yes this should be possible to do, but I've never tried graphite on my own together with metrics.

You can use the MetricCollector provided on the Factory and use the methods like add counter with a name and then you can increment the counter as well. This API is used in the core, but to my knowledge is not (widely) used out there in general, but it should definitely work. If it does not, its a bug.
Comment by Deepu Bhatia [ 11/Sep/14 ]
Hi Michael,
At the bare minimum, I'd at least like to get the definitions for each of the metrics so that there's no misunderstanding. Here's a list of files being created when we use CSV logging.

[MEM] Shutting Down Nodes (NodesToShutdown).csv
[MEM] Response Rate: Success.csv
[MEM] Response Rate: Retry.csv
[MEM] Response Rate: Failure.csv
[MEM] Response Rate: All (Failure + Success + Retry).csv
[MEM] Request Rate: All.csv
[MEM] Reconnecting Nodes (ReconnectQueue).csv
[MEM] Average Time on wire for operations (µs).csv
[MEM] Average Bytes written to OS per write.csv
[MEM] Average Bytes read from OS per read.csv

This is critical to understand to unblock us. I understand that it's not zero overhead. We'll benchmark the performance with PERFORMANCE mode enabled and decide accordingly.
Comment by Deepu Bhatia [ 12/Sep/14 ]
Michael, Matt,
This has become critical for us. We have client side profiling capability now but we don't have an exact definition of the metric. This is a show stopper for us.
Comment by Matt Ingenthron [ 13/Sep/14 ]
Depu: The ones mentioned as "rate" are counters, so you'll need to collect with regularity. The average time and byte counts are just that.

Is there a particular question you're trying to answer? Maybe with that we can give some more specific guidance. I understand it may have to do with latency and where it is? Does the average time on wire correlate to what you're seeing from cbstats timings would be the thing I would be checking for. I'd also be looking for reconnect to be non-zero and retry rate being non-zero.
Comment by Deepu Bhatia [ 16/Sep/14 ]
Hi Matt,
Here's what I am specifically looking for:

[MEM] Shutting Down Nodes (NodesToShutdown).csv - what does this mean?
[MEM] Response Rate: Success.csv - what does this mean?
[MEM] Response Rate: Retry.csv - What retries are this?
[MEM] Response Rate: Failure.csv - What failures does this cover? Is this tracking timeout exceptions?
[MEM] Response Rate: All (Failure + Success + Retry).csv
[MEM] Request Rate: All.csv
[MEM] Reconnecting Nodes (ReconnectQueue).cvs - What is this metric tracking?
[MEM] Average Time on wire for operations (µs).csv - Is this pure network time? Is it in microseconds?
[MEM] Average Bytes written to OS per write.csv
[MEM] Average Bytes read from OS per read.csv





[JCBC-521] VBucketNodeLocator.java getPrimary() and getReplica() log messages are homologous Created: 21/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
VBucketNodeLocator.java has these two methods:
- getPrimary(String k)
- getReplica(String key, int index)

These two write almost the same warning log message when it can't access a node which holds the key.

getPrimary() logs: "for which no server is responsible in the cluster map (-1)."
while
getReplica() logs: "for which no server is responsible in the cluster map."
(without -1 in the message)

 Comments   
Comment by Koji Kawamura [ 21/Aug/14 ]
These two methods log almost the same warning messages, but since those two are indicating different level of severity, those better to be differentiated.
Comment by Michael Nitschinger [ 25/Aug/14 ]
I think it would be okay to push the replica one down to INFO level.Would that be sufficient? I'd rather not change the message itself in a bugfix release.
Comment by Koji Kawamura [ 25/Aug/14 ]
Changing log level of missing replica node to INFO sounds reasonable. Thank you!




Generated at Wed Sep 17 06:14:42 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.