[JCBC-530] Exception when using java client both in Spark and MapReduce Created: 25/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sun Chen Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: Hadoop,Dependencies,HttpCore, Spark,
Remaining Estimate: 40h
Time Spent: Not Specified
Original Estimate: 40h
Environment: Haddop2.4.1,Spark0.13,Java1.7,Ubuntu12.04


 Description   
I am using Couchbase java client(version 1.4.4),I writed some simple mapreaduce job and spark job to write and read data from Couchbase.But I always got an exception.Here is the exception information

-------------------------------------------------------------------------------------------------------------------------------------------------
2014-08-25 13:12:56,048 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V
at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156)
at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266)
at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:31)
at com.eyespage.dc.etl.CouchbaseImportExtractor.extract(CouchbaseImportExtractor.java:20)
at org.apache.sqoop.job.mr.SqoopMapper.run(SqoopMapper.java:96)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
-------------------------------------------------------------------------------------------------------------------------------------------

14/08/09 10:19:32 WARN Utils: Your hostname, precise64 resolves to a loopback address: 127.0.1.1; using 192.168.33.10 instead (on interface eth1) 14/08/09 10:19:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 14/08/09 10:19:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-08-09 10:19:40.569 INFO net.spy.memcached.auth.AuthThread: Authenticated to /127.0.0.1:11210 2014-08-09 10:19:40.677 INFO com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could bootstrap through carrier publication. 2014-08-09 10:19:40.693 INFO com.couchbase.client.CouchbaseConnection: Added {QA sa=localhost/127.0.0.1:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue 2014-08-09 10:19:40.702 INFO com.couchbase.client.CouchbaseClient: CouchbaseConnectionFactory{bucket='Result', nodes=[http://127.0.0.1:8091/pools], order=RANDOM, opTimeout=2500, opQueue=16384, opQueueBlockTime=10000, obsPollInt=10, obsPollMax=500, obsTimeout=5000, viewConns=10, viewTimeout=75000, viewWorkers=1, configCheck=10, reconnectInt=1100, failureMode=Redistribute, hashAlgo=NATIVE_HASH, authWaitTime=2500} Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.protocol.RequestUserAgent.<init>(Ljava/lang/String;)V at com.couchbase.client.ViewConnection.<init>(ViewConnection.java:156) at com.couchbase.client.CouchbaseConnectionFactory.createViewConnection(CouchbaseConnectionFactory.java:254) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:266) at com.couchbase.client.CouchbaseClient.<init>(CouchbaseClient.java:194) at com.eyespage.dc.etl.c2s.Run$.main(main.scala:16) at com.eyespage.dc.etl.c2s.Run.main(main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2014-08-09 10:19:40.836 INFO net.spy.memcached.auth.AuthThread: Authenticated to localhost/127.0.0.1:11210 - See more at: http://www.couchbase.com/communities/q-and-a/exception-when-using-scala-call-java-sdk%EF%BC%8Cand-run-spark#sthash.CmL5dcRz.dpuf








[JCBC-532] Implement Common Flags Created: 27/Aug/14  Updated: 27/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Brett Lawson Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Implement the Common Flags formatting as defined by the specification.




[JCBC-519] Timeout Not Honored on First Call to Java asyncGetBulk Created: 11/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: Core
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File App.java    
Issue Links:
Duplicate

 Description   
This was observed on Couchbase Server 2.2.0. The same behavior is not evident on Couchbase Server 2.5.1. It appears to be independent of any Java client version. The very first call of asyncGetBulk does not respect the timeout value, and continues to process and wait. Subsequent calls do properly respect the timeout value. One interesting note is that this only occurs when the timeout value is "close" to the actual execution time. That is, a very short timeout is indeed honored.

Steps to reproduce:

* Download a Java client, such as http://packages.couchbase.com/clients/java/1.4.3/Couchbase-Java-Client-1.4.3.zip

* Unzip

* Copy the attached App.java into the unzipped directory

* Compile and run:

javac -cp couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar App.java

java -cp .:couchbase-client-1.4.3.jar:spymemcached-2.11.4.jar:jettison-1.1.jar:netty-3.5.5.Final.jar:commons-codec-1.5.jar:httpcore-4.3.jar:httpcore-nio-4.3.jar App > 143.txt

* Note the following output in the first group. It would be expected that the first call would timeout, but does not:

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=108ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=16ms


Exception:********************
Timeout future is: true
Operation timed out. - failing nodes: 10.4.2.107/10.4.2.107:11210, cbhost.domain/10.4.2.57:11210
Timeout set to:16
getMulti : Time taken=18ms

Timeout future is: false
Fetched 72 keys
Timeout set to:16
getMulti : Time taken=17ms


ASK - Determine why the first call does not honor the timeout, and ideally suggest a workaround for Couchbase Server 2.2.0, short of upgrading to Couchbase Server 2.5.1 (where the issue does not appear exist, and therefore looks to be a server-side issue)




 Comments   
Comment by Aleksey Kondratenko [ 11/Aug/14 ]
Not honoring timeouts cannot be server side issue. Passing to Michael who owns couchbase java sdk
Comment by Jeff Dillon [ 12/Aug/14 ]
Please disregard my previous comments with respect to being apparently fixed with Couchbase Server 2.5.1. This behavior evidences itself across all CB server versions, and also reproduces with CB Java client 1.4.4. In my latest 2.5.1 test, I set the timeout to 30 ms, and see the same incorrect behavior. So this is indeed a client issue (as noted), apologies on the initial misinformation.
Comment by Michael Nitschinger [ 14/Aug/14 ]
A few remarks:

- The simple benchmark also profiles logging and so forth.
- System.currentTimeMillis() precision is sometimes beyond all joke, always use nanotime.

So something like this is more accurate:

        while (low == 10) {
            try {
                startTime = System.nanoTime();
                future = client.asyncGetBulk(Main.getKeys());
                future.get(timeout, TimeUnit.MILLISECONDS);
                endTime = System.nanoTime();
                System.out.println("Timeout future is: " + future.isTimeout());
                System.out.println("Fetched " + Main.getKeys().size() + " keys");
            } catch (Exception e) {
                System.out.println("\nException:********************\nTimeout future is: " + future.isTimeout());
                System.out.println(e.getMessage());
            }
            operationTime = endTime - startTime;
            System.out.println("Timeout set to:" + timeout + "\ngetMulti : Time taken=" + TimeUnit.NANOSECONDS.toMillis(operationTime) + "ms\n");
        }

Now also note that profiling on the blocking .get() call not only waits on the latch where the timeout is applied, but it also puts all data together and so forth. So as far as I can see it the timeout is honored, its just not what you benchmark that close. Keep in mind that the first few calls are subject t JVM warmup, optimizations and so forth which can take some time. It's not a huge surprise that the first few calls are slightly slower, but I guess thats not what the JVM optimizes for compared to a long-running server.

Does that help?
Comment by Jeff Dillon [ 14/Aug/14 ]
I am seeing the same behavior when using nanoTime. And it's not slightly slower, it's an order of magnitude difference. And I don't think it's JVM warmup either, as different timeout values show consistent differences every time. Using a short time out value works as expected, but is not the timeout value they desire. When the timeout is "close" to the actual execution time, it experiences this behavior. This is what I'm seeing in my tests, and is what the customer is seeing also.
Comment by Matt Ingenthron [ 20/Aug/14 ]
Jeff Dillon: I understand from James this has been observed on 2.5.1 now as well. Can you confirm and update it appropriately? Also, I've moved it to JCBC, since the investigation has lead here.
Comment by Jeff Dillon [ 20/Aug/14 ]
Yes, please see my previous comment from 12/Aug/14 1:47 PM




[JCBC-520] Running SSL Test on 2.0 Client hangs when certificate is not valid Created: 20/Aug/14  Updated: 20/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 2.0-dp3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Wei-Li Liu Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
The SSL test I make can be seen from
https://github.com/weilliu/couchbase-java-client/commit/97c3144692627d6e657a0cc157f5f9aa644de0f2
http://review.couchbase.org/#/c/40763/

Currently if we enable ssl, and when it hits a certificate which is not valid/not exist, the test would hang


11:38:16.868 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STARTED
11:38:17.843 [DEBUG] [TestEventLogger]
11:38:17.843 [DEBUG] [TestEventLogger] com.couchbase.client.java.SSLTest > ClientCertificateRefresh STANDARD_OUT
11:38:17.843 [DEBUG] [TestEventLogger] File copied!
11:38:17.844 [DEBUG] [TestEventLogger] Reconnect with the Server
> 11:39:06.2 [ DEB] [org.gradle.process.internal.ExecHandleRunner] Abort requested. Destroying process: Gradle Worker 1.
> Building 85% > :integrationTest > 29 tests completed, 29 skippedWei-Lis-MacBook-Pro:couchbase-java-client wei-li$






[JCBC-536] Interpretation and Performance Impact of Client Profiling Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Task Priority: Major
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Customer is using client side profiling as suggested in:

http://blog.couchbase.com/whats-new-couchbase-java-sdk-12

Do we have documentation on the Meters, Counters and Histogram output? What would the performance impact be, if any? Can this be safely left in place in a Production environment, for example?




[JCBC-521] VBucketNodeLocator.java getPrimary() and getReplica() log messages are homologous Created: 21/Aug/14  Updated: 25/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Minor
Reporter: Mark Woosey Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
VBucketNodeLocator.java has these two methods:
- getPrimary(String k)
- getReplica(String key, int index)

These two write almost the same warning log message when it can't access a node which holds the key.

getPrimary() logs: "for which no server is responsible in the cluster map (-1)."
while
getReplica() logs: "for which no server is responsible in the cluster map."
(without -1 in the message)

 Comments   
Comment by Koji Kawamura [ 21/Aug/14 ]
These two methods log almost the same warning messages, but since those two are indicating different level of severity, those better to be differentiated.
Comment by Michael Nitschinger [ 25/Aug/14 ]
I think it would be okay to push the replica one down to INFO level.Would that be sufficient? I'd rather not change the message itself in a bugfix release.
Comment by Koji Kawamura [ 25/Aug/14 ]
Changing log level of missing replica node to INFO sounds reasonable. Thank you!




[JCBC-537] Setting Reader/Writer Worker value on Server causes ParseException on Client Created: 29/Aug/14  Updated: 29/Aug/14

Status: Open
Project: Couchbase Java Client
Component/s: None
Affects Version/s: 1.4.4
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Jeff Dillon Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File output.log    
Issue Links:
Duplicate

 Description   
When setting the "Number of suggested reader/writer workers" in the Admin UI to 8, the 1.4.4 client then begins to see ParseException errors during the bootstrap phase. The client appears to continue as expected after that, but the error is concerning. The exception does not occur in the .NET or C client libraries.

To repro:

* Set the worker value to 8 in the UI
* Run the HelloWorld example from:

http://www.couchbase.com/communities/java/getting-started

Relevant code is:

// (Subset) of nodes in the cluster to establish a connection
    List<URI> hosts = Arrays.asList(
      new URI("http://127.0.0.1:8091/pools")
    );
 
    // Name of the Bucket to connect to
    String bucket = "default";
 
    // Password of the bucket (empty) string if none
    String password = "";
 
    // Connect to the Cluster
    CouchbaseClient client = new CouchbaseClient(hosts, bucket, password); // <- exception here

When running the example, the response is consistently:

2014-08-29 15:36:21.979 INFO net.spy.memcached.auth.AuthThread: Authenticated to /<ip_address>:11210
2014-08-29 15:36:22.014 WARN com.couchbase.client.vbucket.provider.BucketConfigurationProvider: Could not parse config, retrying bootstrap.
java.text.ParseException: A JSONObject text must begin with '{' at character 0 of
at com.couchbase.client.vbucket.config.ConfigurationParserJSON.parseBucket(ConfigurationParserJSON.java:148)



 Comments   
Comment by Matt Ingenthron [ 29/Aug/14 ]
If you crank up the log level, it should print out what the cluster is returning. Maybe the .NET and C clients are probably seeing the same behavior but not logging the warning?
Comment by Jeff Dillon [ 29/Aug/14 ]
In .NET I used the following log settings, and didn't see the error:

      <priority value="ALL" />
      <level value="DEBUG" />
Comment by Matt Ingenthron [ 29/Aug/14 ]
Yep, might be a bug that .NET doesn't log a warning on bad response and goes on to the next node.

The parse error comes from Jettison. https://github.com/couchbase/couchbase-java-client/blob/669c9580674082b2acccd769d9da5e1b3bb3e417/src/main/java/com/couchbase/client/vbucket/config/ConfigurationParserJSON.java#L148

It'd be really helpful if you could get the log to tell us what the cluster is returning.
Comment by Jeff Dillon [ 29/Aug/14 ]
Attaching Java log using:

logging.properties
handlers = java.util.logging.FileHandler
java.util.logging.FileHandler.pattern=output.log
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
com.couchbase.client.vbucket.level = FINEST
com.couchbase.client.vbucket.config.level = FINEST
com.couchbase.client.level = FINEST




Generated at Sat Aug 30 12:19:58 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.