Using couchbaseclient inside netty server
This bug applies to CouchbaseClient java library. I can't seem to find a way to use Couchbase client inside Netty server.
Using Couchbase client without Netty works fine. However, if the application is embedding Netty, the couchbase client fails to instantiate a client. I see that CouchbaseClient is using Netty inside it.
The application when running inside Netty is running inside a Netty worker thread. The thread tried to create a new CouchbaseClient which uses Netty and calls await on a channel. Netty throws an IllegalStateException noticing that the thread is in IO thread (Netty server Worker Thread).
If you have any suggestions for workaround, please let me know. Thanks :)
I am not sure if the bug should be posted to Couchbase or to Netty, but here is the stack trace.
java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.
at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:296) ~[netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:208) ~[netty-3.2.0.Final.jar:na]
at com.couchbase.client.vbucket.BucketMonitor.createChannel(BucketMonitor.java:209) ~[couchbase-client-1.0.3.jar:1.0.3]
at com.couchbase.client.vbucket.BucketMonitor.startMonitor(BucketMonitor.java:177) ~[couchbase-client-1.0.3.jar:1.0.3]
at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:261) ~[couchbase-client-1.0.3.jar:1.0.3]
at com.couchbase.client.CouchbaseClient.(CouchbaseClient.java:158) ~[couchbase-client-1.0.3.jar:1.0.3]
at com.couchbase.client.CouchbaseClient.(CouchbaseClient.java:125) ~[couchbase-client-1.0.3.jar:1.0.3]
at com.couchbase.client.CouchbaseClient.(CouchbaseClient.java:77) ~[couchbase-client-1.0.3.jar:1.0.3]
[Some Application Stack Here]
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:97) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:513) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:497) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:434) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:340) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:272) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.0.Final.jar:na]
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) [netty-3.2.0.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_07]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_07]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_07]
I worked around the bug by using the following code
public class CouchClientProvider {
private static final ExecutorService EXECUTOR_SERVICE = Executors.newCachedThreadPool();
public static CouchbaseClient get() throws Exception {
Future future = EXECUTOR_SERVICE.submit(new CouchClientWorker());
return future.get();
}
static class CouchClientWorker implements Callable {
public CouchbaseClient call() throws Exception {
return new CouchbaseClient(new LinkedList(), "","");
}
}
}
This essentially moves the Couchbase client creation to a separate thread and stops netty from complaining.
Thanks for pointing this out and including a fix and a workaround. If you get a chance to put in a fix for the client that would be great and I am happy to help you out with the contrib process. If not I have filed a bug here and one of our developers will get around to fixing it as soon as possible.
On further research, I found that the problem lies with CouchbaseClient
com.couchbase.client.vbucket.BucketMonitor class, function createChannel(), line 209 (version 1.0.3)
// Wait until the connection attempt succeeds or fails.
channel = future.awaitUninterruptibly().getChannel();
If the code is modified to add a listener using future.addListener(), that would get rid of this Netty exception and then the code can be used inside an application using Netty server too.
I will try to build a patch and learn about Couchbase contrib process. But, if you are familiar with codebase and can fix it soon, I would very much appreciate that.
Thanks.