The NoClassDef confuses me… In what environment are you running your code?
I am running in eclipse, jdk 7 (32 bit or 64 bit I get the same error). With windows 7 professional 64 bit. I have some sample code with api 1.2.2, and 2 with 2.0.3 (one sharing cluster+ucket and one not). What I have noticed is only the 1.2.2 can handle lots of load the 2.0.3 gets errors when constantly call in a loop - even without multiple threads calling couchbase at the same time. With 1.2.2 I can call any number of loop iterations ( I used 3000) and they all completed. With 2.0.3 I can not get even 200 to complete most of the time.
I put the basic files I am using to talk to couchbase here https://github.com/racarlson/temp2 , and the thread I am using to test (I added a syncrhonize block because without it no CacheManager.java version works without using lots of memory). I have 3 versions of CacheManager as stated above, they start with CacheManager_java in that github. The 1.2.2 version with synchronized block thread works fine, the 2.0.3 do not. There might be a memory leak in 2.0.3 since closing the cluster and bucket after each call still seems to give me OutOfMemoryErrors. In eclipse I am using visualvm with arguments -Xms1180m -Xmx1180m for memory. In production we planned on having a lot larger load that 1000-3000 so if I can’t handle 3000 with 1GB that no use raising the memory even more. There is also properties file being read there for 1.2.2 or 2.0.3, which is basically just the bucket name and url for couchbase
Hello @daschl,
I am new to Couchbase and facing TimeOutException when trying to open two buckets.
My java code is as below
public CouchbaseBucketFactory(String nodeList,int connectionTimeout) {
CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
.connectTimeout(connectionTimeout)
.queryEnabled(true).build();
List<String> nodes = Arrays.asList(nodeList.split(FarebrokerConstants.COMMA_STRING));
Cluster cluster = CouchbaseCluster.create(env,nodes);
Bucket b2 = cluster.openBucket("Bucket1");
Bucket b1 = cluster.openBucket("Bucket2");
System.out.println("Bucket is opened");
}
The input to the above constructor method is
nodeList = 192.168.158.6,192.168.158.11
connectionTimeout = 10000
The code is working perfect when only one bucket is opened. Issue comes when the second bucket is tried to open.
- Jdk version is 1.6.
- Tomcat 6 is the web server.
- java-client 2.1.4 jar is used
- couchbase-server-community-3.0.1-centos5.x86_64.rpm is used for couchbase installation
Trace
Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:93)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:108)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:99)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:89)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:79)
at com.his.farebroker.interfaces.couchbase.CouchbaseBucketFactory.(CouchbaseBucketFactory.java:79)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126)
… 94 more
@CouchbaseTrial were you by any chance trying to open a bucket that doesn’t exist first? (ie. first bucket opening fails)
We’ve come across a bug where if first openBucket attempt fails, following attempts (even on different buckets) will timeout. This has been corrected in the newest release, 2.2.2.
Unfortunately, I’m using version 2.2.4 and this bug still seems to be present. It doesn’t always occur, but more often than not, it does. Specifically, when I try to open a second bucket the following exception is thrown:
java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:296)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:271)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:261)
My code looks like this:
// Get a connection to the DB
cluster = CouchbaseCluster.create( “couchbasehost” );
if( cluster != null )
{
idBucket = cluster.openBucket( “ID_Repository”, 30, TimeUnit.SECONDS );
topicBucket = cluster.openBucket( “topics”, 30, TimeUnit.SECONDS );
messageBucket = cluster.openBucket( “topicMessages”, 30, TimeUnit.SECONDS );
}
At the line where I try to open the topicBucket, the above exception is thrown.
Any thoughts, workarounds, etc will be appreciated.
@kdolan would you be able to provide a trace-level log of the failure? Any other context (eg. does the first bucket actually exist, etc…)?
Simon,
Thanks for the reply.
First, to answer your specific question, yes, the first bucket does exist.
A little background: I’ve encountered this problem in two places. One is our application. The snippet of code I included in my original message is from there. The second place is a utility I wrote to automate the creation of buckets and views on a Couchbase instance (we have quite a few test servers set up and doing it manually gets very tiresome). In that case, the code connects to the cluster then creates a bucket and its views, closes the bucket, then immediately repeats with another bucket. It usually works, but once-in-while the same problem occurs when it tries to open the second bucket.
In our application, I created a workaround after I sent you my original message. Basically, only one bucket is open at any given time. A bucket is acquired, used, then closed in a finally block before another bucket is opened. So far, this seems to work. I don’t know what the cost of opening and closing buckets is, but it is a little awkward.
I initially encountered the timeout problem last Thursday and it was pretty consistent. This morning I recreated the problem by commenting out some of my workaround code and repeating the former scenario. I’m noticing that the error happens on some of the test machines, but not others. (To be honest, I’m not sure if this was the case earlier or not.) I have pinged the machines where the timeout occurs, but there doesn’t appear to be any connection issues. Each Couchbase instance runs in its own VM on one of two servers.
I set up the logging per the instructions on your website, but it doesn’t seem to have had any effect on the output. Below is the output I’m getting. The exception is being thrown after the 30 second timeout occurs. However, note the very last line of the output. The topics bucket does eventually open even though the timeout occurred and the code exited the constructor where the buckets are opened.
INFO | Connected to Node couchbasehost
INFO | Opened bucket ID_Repository
Feb 16, 2016 11:23:48 AM com.soliel.topics.TopicManager
SEVERE: Error contructing TopicManager: java.util.concurrent.TimeoutException
java.lang.RuntimeException: java.util.concurrent.TimeoutException
at com.couchbase.client.java.util.Blocking.blockForSingle(Blocking.java:75)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:296)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:271)
at com.couchbase.client.java.CouchbaseCluster.openBucket(CouchbaseCluster.java:261)
at com.soliel.topics.TopicManager.(TopicManager.java:113)
at com.soliel.topics.TopicManager.(TopicManager.java:75)
at com.soliel.routing.ControlProcessor.(ControlProcessor.java:41)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:379)
at org.apache.camel.util.ObjectHelper.newInstance(ObjectHelper.java:1537)
at org.apache.camel.util.ReflectionInjector.newInstance(ReflectionInjector.java:32)
at org.apache.camel.impl.DefaultInjector.newInstance(DefaultInjector.java:45)
at org.apache.camel.component.bean.RegistryBean.getBean(RegistryBean.java:89)
at org.apache.camel.component.bean.RegistryBean.createCacheHolder(RegistryBean.java:60)
at org.apache.camel.model.BeanDefinition.createProcessor(BeanDefinition.java:251)
at org.apache.camel.model.ProcessorDefinition.makeProcessorImpl(ProcessorDefinition.java:534)
at org.apache.camel.model.ProcessorDefinition.makeProcessor(ProcessorDefinition.java:495)
at org.apache.camel.model.ProcessorDefinition.addRoutes(ProcessorDefinition.java:219)
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:1069)
at org.apache.camel.model.RouteDefinition.addRoutes(RouteDefinition.java:196)
at org.apache.camel.impl.DefaultCamelContext.startRoute(DefaultCamelContext.java:944)
at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:3245)
at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:2968)
at org.apache.camel.impl.DefaultCamelContext.access$000(DefaultCamelContext.java:172)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2799)
at org.apache.camel.impl.DefaultCamelContext$2.call(DefaultCamelContext.java:2795)
at org.apache.camel.impl.DefaultCamelContext.doWithDefinedClassLoader(DefaultCamelContext.java:2818)
at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:2795)
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61)
at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:2764)
at com.soliel.Application.(Application.java:88)
at com.soliel.Application.main(Application.java:173)
Caused by: java.util.concurrent.TimeoutException
… 35 more
INFO | AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
INFO | StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at Stream caching :: Apache Camel
INFO | Jetty JMX Extensions is enabled
INFO | Using default Jetty continuation timeout for: Endpoint[jetty:http://10.73.21.12:8182/tc-service/request]
WARN | No Server set for org.apache.camel.component.jetty.JettyHttpComponent$1@482bc000
INFO | Route: route1 started and consuming from: Endpoint[jetty:http://10.73.21.12:8182/tc-service/request]
INFO | Route: route2 started and consuming from: Endpoint[jetty:http://10.73.21.12:8182/tc-service/callback]
INFO | Route: route3 started and consuming from: Endpoint[jetty:http://10.73.21.12:8182/tc-service/control]
INFO | Route: route4 started and consuming from: Endpoint[jetty:http://10.73.21.12:8182/tc-service/topic]
INFO | Total 4 routes, of which 4 is started.
INFO | Apache Camel 2.16.0 (CamelContext: camel-1) started in 33.379 seconds
INFO | Successfully connected to tcp://10.73.25.12:61616
PeerConsumer: connection to ae0ab44c-bd88-43a0-912e-c7cfdc5e11c7 started
INFO | Opened bucket topics
At this point, I’m not sure where the problem lies. Any input you can give will be appreciated.
kdolan
Further to my previous message:
We have two servers set up for testing, each with a number of VMs running Couchbase. The older server uses Couchbase 2.2 and on it I can open multiple buckets without any problem. On the newer server, the VMs run Couchbase 4.0. For these I observe the following:
- The first bucket opens normally (takes about 1 second)
- All subsequent buckets take 75 seconds each to open.
The timing is suspicious: almost exactly the same each VM, varying by only 50 MS at most.
I hope this helps.
kdolan
That’s interesting. Concerning logs, you should see much larger ones with dumps of network packets, so you’ve probably missed a step. Did you add a supported logging library to your classpath and configured it so that it logs at TRACE level?
I’m curious to know how are these VMs sized, which OS are they using, and what is the configuration of the different buckets? Where is the client executed (different machine, which OS?)
If you run a simple program that absolutely just creates the Cluster then opens the 3 Buckets, what happens?
Simon,
I’m a developer using the systems. I didn’t set them up, but I’ll try to answer your questions as best I can.
The VMs on both servers use Centos 6.2, 64 bit. All VMs have the same resource allocation: 1 CPU and 2 GB of memory. In each VM, Couchbase is the only application running; all queries to the Couchbase instances come from Java clients located in other VMs, but on the same server.
I do not believe there is anything special about the bucket configurations and they are basically identical. All have about the same amount of memory allocated: 300-400 MB. That may seem small, but these are just development environments and the data stored in them is very minimal.
At this point I can’t do any further testing because the decision was made to remove Couchbase 4.0 on the three VMs in question and replace it with 2.2. Having done that, everything is now working correctly and the buckets open without any problems.
All I can do is clearly reiterate the issue encountered: Using the 2.2.4 version of the Java client with Couchbase 4.0, the first bucket opens normally. Bucket 2 takes 75 seconds to open. Bucket 3 takes 75 seconds to open. The 75 seconds is consistent across VMs, so it implies that some timing mechanism might be involved.
Even though we have rolled back to Couchbase 2.2, I hope that some answer is found, because we will eventually have to move forward.
Thanks for your help.
kdolan
@simonbasle,
While defining CouchbaseEnvironment we were using queryEnabled(true). We removed this and error also gone now. (couchbase-server-community-3.0.1-centos5.x86_64.rpm is used for couchbase installation)
CouchbaseEnvironment env = DefaultCouchbaseEnvironment.builder()
.connectTimeout(connectionTimeout)
.queryEnabled(true).build();