com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight

amit · October 27, 2016, 7:02pm

Hi Couchbase Java SDK Client experts, we are facing the following problem in some of our environments, when we deploy the application that is using Couchbase JAVA sdk client and the Couchabse server on the same VM. Following are some details about the deployment before posting the exception that is logged by Couchbase client.

O.S. : CentOS release 6.6 (Final)
Couchbase Server Version: 3.0.1 Community Edition (build-1444)
Couchbase Client SDK Version: couchbase-java-client-2.1.4.jar, rxjava-1.0.4.jar
Number of Nodes: 1
No specific configuration is passed to the Couchbase client regarding any timeouts so it should be using the default one. And both Couchbase server and the application are on the same VM.

We are seeing the following message logged every minute

2016-10-26 11:04:32,419  INFO AbstractGenericHandler:? - Exception thrown while cancelling outstanding operation: KeepAliveRequest{observable=rx.subjects.AsyncSubject@209c991c, bucket='null'}
rx.exceptions.OnErrorNotImplementedException: Request cancelled in-flight.
	at rx.Observable$31.onError(Observable.java:7134)
	at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:154)
	at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:111)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:228)
	at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262)
	at com.couchbase.client.core.endpoint.query.QueryHandler.handlerRemoved(QueryHandler.java:612)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	... 16 more
2016-10-26 11:04:32,424  INFO AbstractGenericHandler:? - Exception thrown while cancelling outstanding operation: KeepAliveRequest{observable=rx.subjects.AsyncSubject@7cca33c1, bucket='null'}
rx.exceptions.OnErrorNotImplementedException: Request cancelled in-flight.
	at rx.Observable$31.onError(Observable.java:7134)
	at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:154)
	at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:111)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:228)
	at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:262)
	at com.couchbase.client.core.endpoint.query.QueryHandler.handlerRemoved(QueryHandler.java:612)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:798)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:767)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:759)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:743)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$6.run(AbstractChannel.java:615)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	... 16 more
	
	
	
2016-10-26 11:18:40,157  INFO AbstractGenericHandler:? - [our_host_name_here/our_ip_here:8093][QueryEndpoint]: Connection reset by peer: Connection reset by peer
2016-10-26 11:18:40,157  INFO AbstractGenericHandler:? - Exception thrown while cancelling outstanding operation: KeepAliveRequest{observable=rx.subjects.AsyncSubject@3fd05d52, bucket='null'}
rx.exceptions.OnErrorNotImplementedException: Request cancelled in-flight.
	at rx.Observable$31.onError(Observable.java:7134)
	at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:154)
	at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:111)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:228)
	at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:253)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:137)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:834)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:161)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	... 21 more
2016-10-26 11:18:40,158  INFO AbstractGenericHandler:? - Exception thrown while cancelling outstanding operation: KeepAliveRequest{observable=rx.subjects.AsyncSubject@24dcd984, bucket='null'}
rx.exceptions.OnErrorNotImplementedException: Request cancelled in-flight.
	at rx.Observable$31.onError(Observable.java:7134)
	at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:154)
	at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:111)
	at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onError(SubjectSubscriptionManager.java:228)
	at rx.subjects.AsyncSubject.onError(AsyncSubject.java:115)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:253)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:137)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:834)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:161)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	... 21 more

Another interesting point here is that this does not happen if I deploy the same application and Couchbase on my laptop. So can somebody give some pointers where to look for resolving this problem on our servers. Can it be fixed by some Couchbase configuration or is there any OS related settings that we need to look into.

subhashni · November 1, 2016, 1:36pm

Hi @amit, There was a bug that got fixed which was causing this, JVMBC-340, Can you upgrade to the latest client 2.3.4 and check if the problem is resolved?

Thanks.

amit · November 3, 2016, 5:02pm

Hi Subhashni,

Thank you very much for your answer. Is it ok to use the latest java client with Couchbase Server Version: 3.0.1 Community Edition. Can it cause any unknown side-effects?

subhashni · November 3, 2016, 5:29pm

Yes it supports server version 3.x

amit · November 3, 2016, 8:33pm

Thanks Subhashni, I will give it a try and update here if it resolves the problem.

Amit

debon.julien · November 16, 2016, 12:55pm

Hello,

I have exactly the same problem.
I use org.springframework.data:spring-data-couchbase:jar:2.1.5.RELEASE which is based on com.couchbase.client:java-client:jar:2.2.8 and I have the same issue.

I tried to force com.couchbase.client:java-client:jar:2.3.5 by declaring it as direct dependency of my project with Maven, but another (similar) issue appears:

WARN 2016-11-16 13:53:00.820 com.couchbase.client.core.endpoint.AbstractGenericHandler - [localhost/127.0.0.1:8093][QueryEndpoint]: Got error while consuming KeepAliveResponse.
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:502)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:483)
at com.couchbase.client.core.endpoint.query.QueryHandler.handlerRemoved(QueryHandler.java:746)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:616)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:868)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:826)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:43)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1276)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:185)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:171)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:811)
at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:699)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:402)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)

I would greatly appreciate any help to solve this log spam (functionally everything seems to work fine).

daschl · November 16, 2016, 6:27pm

@debon.julien do you have port 8093 open to all the query nodes and is there nothing in between thats causing trouble? it looks like the socket got closed while the SDK is doing heartbeats against it.

debon.julien · November 17, 2016, 8:49am

I run Couchbase via a Docker container (I pass the -p 8093:8093 parameter to forward port 8093).
I think the port 8093 is correctly open:

| => lsof -i4TCP:8093 | grep LISTEN
com.docke 29373 Julien 456u IPv4 0x17a9fb6f621e707f 0t0 TCP *:8093 (LISTEN)

Please let me know if there’s any other diagnostic I can run.

subhashni · November 17, 2016, 7:17pm

@debon.julien Is there any previous logs that indicate a connection close before the KeepAlive response error?

debon.julien · November 18, 2016, 9:57am

I restarted my docker container and the problem seems to have disappeared. Maybe it somehow got into a weird state… I’ll update if it happens again.

preethi.minti · July 27, 2017, 4:40am

We had the same issue in production few days ago and had to restart our Wildfly instance. This happened after being in production for 50 days. We did not have any deployments what so ever. We using CouchbaseClient jar 2.3.0. Is this a known issue, is there a fix for this or do we know why it happens?

ingenthr · July 27, 2017, 6:01am

For the keepalive @preethi.minti, we just added in 2.4.7 a change that will proactively rebuild the connections if there is a network failure that causes connections to go half open. It’s hard to say if this is the same issue in your case as the thread covers a couple different things and your logs aren’t clear, but if it’s related to something where a client would end up half-open, 2.4.7 should help.

esha.jagdale · September 28, 2017, 5:36am

Hi Couchbase Team,
We are facing the same issue on our production Couchbase 4.6.3 running N1QL query with Java SDK 2.1. Is there any work around until we upgrade our SDK to 2.4.7? Since its our production system, we cannot immediately upgrade SDK. Here are the logs for reference:

Application logs:
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:281) ~[core-io-1.1.3.jar:1.1.3]
at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:253) ~[core-io-1.1.3.jar:1.1.3]
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:246) ~[core-io-1.1.3.jar:1.1.3]
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:224) ~[core-io-1.1.3.jar:1.1.3]
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[core-io-1.1.3.jar:1.1.3]
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:137) ~[core-io-1.1.3.jar:1.1.3]

Couchbase logs:
Service ‘query’ exited with status 1. Restarting. Messages: runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc20932ffe8 sp=0xc20932ffe0
created by github.com/couchbase/indexing/secondary/queryport/n1ql.func·007
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/n1ql/secondary_index.go:867 +0xcfe
[goport] 2017/09/28 10:12:38 /opt/couchbase/bin/cbq-engine terminated: signal: aborted

ismail.iqbal.ap · June 12, 2018, 3:04pm

Hi Couchbase Team,
I am using couchbase 5.1 with Java SDK 2.5. I am getting the Request cancelled in-flight while querying the view by start key and end key in my local workspace.

Error:
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:551)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:532)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:624)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:876)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:842)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:44)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1284)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:176)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:162)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:819)
at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:717)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Unknown Source)
2018-06-12 09:55:25 DEBUG CarrierRefresher:105 - Error while refreshing bucket config, ignoring.
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:551)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:532)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:624)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:876)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:842)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:44)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1284)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:176)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:162)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:819)
at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:717)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Unknown Source)

daschl · June 12, 2018, 3:15pm

@ismail.iqbal.ap can you share more logs? your request got cancelled because it was in-flight and the underlying socket got closed at the same time. Check the errors above / around to figure out the root cause.

ismail.iqbal.ap · June 12, 2018, 3:20pm

Hi daschl,
As I am getting this error randomly, I lost my logs. I will post it again as soon as i get it again. Thanks

ismail.iqbal.ap · June 13, 2018, 6:31pm

Hi,
After my testcase yields the result, its throwing Request cancelled in-flight error
Test case:
@After
public void shutdown() throws CacheException {
cluster.disconnect();
}

@Test
public void getRITest() throws Exception {
try {
Bucket bucket = getBucket(“RI”);
Object obj = bucket.get(“0001BA74414CEB”);
Assert.assertNotNull(obj);
logger.debug("@@@@@@@@@@ RI: " + obj.toString());
} catch (Exception e) {
e.printStackTrace();
}
}

Error Logs:
2018-06-13 12:18:58 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Adding Service CONFIG
2018-06-13 12:18:58 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Service CONFIG already added, skipping.
2018-06-13 12:18:59 INFO CouchBaseProvider:278 - Cache hit for key: 0001BA74414CEB
2018-06-13 12:18:59 DEBUG ConfigurationProvider:85 - Received signal to proactively refresh (a maybe outdated) configuration.
2018-06-13 12:18:59 DEBUG CouchBaseProviderTest:149 - @@@@@@@@@@ RI: JsonDocument{id=‘0001BA74414CEB’, cas=1467986550729605120, expiry=0, content={“timezone”:“UTC”,“source”:“Web”,“locale”:“en_US”,“userName”:“Ismail Iqbal”}}
2018-06-13 12:18:59 DEBUG ConfigurationProvider:85 - Closing all open buckets
2018-06-13 12:18:59 DEBUG ConfigurationProvider:90 - Closing bucket request-instance
2018-06-13 12:18:59 DEBUG ConfigurationProvider:90 - Removing bucket request-instance configuration from known configs.
2018-06-13 12:18:59 DEBUG RequestHandler:85 - Got notified of a new configuration arriving.
2018-06-13 12:18:59 DEBUG RequestHandler:85 - Starting reconfiguration.
2018-06-13 12:18:59 DEBUG RequestHandler:85 - No open bucket found in config, disconnecting all nodes.
2018-06-13 12:18:59 DEBUG RequestHandler:90 - Got instructed to remove Node NetworkAddress{tparheaodd033.enterprisenet.org/10.8.48.137, fromHostname=false, reverseDns=true}
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Instructing Service SEARCH to disconnect.
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][SearchService]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Service:90 - [tparheaodd033.enterprisenet.org][SearchService]: New number of endpoints is 0
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Instructing Service QUERY to disconnect.
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][QueryService]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Service:90 - [tparheaodd033.enterprisenet.org][QueryService]: New number of endpoints is 0
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Instructing Service VIEW to disconnect.
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][ViewService]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Service:90 - [tparheaodd033.enterprisenet.org][ViewService]: New number of endpoints is 0
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Instructing Service CONFIG to disconnect.
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][ConfigService]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Node:85 - [10.8.48.137/tparheaodd033.enterprisenet.org]: Instructing Service BINARY to disconnect.
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][KeyValueService]: Got instructed to disconnect.
2018-06-13 12:18:59 DEBUG Service:90 - [tparheaodd033.enterprisenet.org][KeyValueService]: New number of endpoints is 1
2018-06-13 12:18:59 DEBUG Service:85 - [tparheaodd033.enterprisenet.org][KeyValueService]: Disconnecting Endpoint during Service disconnect.
2018-06-13 12:18:59 DEBUG AbstractGenericHandler:85 - [tparheaodd033.enterprisenet.org/10.8.48.137:11210][KeyValueEndpoint]: Channel Inactive.
2018-06-13 12:18:59 INFO Endpoint:115 - [tparheaodd033.enterprisenet.org/10.8.48.137:11210][KeyValueEndpoint]: Got notified from Channel as inactive, attempting reconnect.
2018-06-13 12:18:59 DEBUG AbstractGenericHandler:85 - [tparheaodd033.enterprisenet.org/10.8.48.137:11210][KeyValueEndpoint]: Cancelling 1 outstanding requests.
2018-06-13 12:18:59 INFO ConfigurationProvider:115 - Closed bucket request-instance
2018-06-13 12:18:59 DEBUG Endpoint:85 - [tparheaodd033.enterprisenet.org/10.8.48.137:11210][KeyValueEndpoint]: Disconnected Endpoint.
2018-06-13 12:18:59 INFO Node:115 - Disconnected from Node 10.8.48.137/tparheaodd033.enterprisenet.org
2018-06-13 12:18:59 DEBUG ConfigurationProvider:85 - Shutting down ConfigurationProvider.
2018-06-13 12:18:59 DEBUG Node:85 - Disconnected (DISCONNECTING) from Node NetworkAddress{tparheaodd033.enterprisenet.org/10.8.48.137, fromHostname=false, reverseDns=true}
2018-06-13 12:18:59 DEBUG CarrierRefresher:105 - Could not fetch config from bucket “request-instance” against “NetworkAddress{tparheaodd033.enterprisenet.org/10.8.48.137, fromHostname=false, reverseDns=true}”.
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:551)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:532)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:624)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:876)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:842)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:44)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1284)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:176)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:162)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:819)
at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:717)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Unknown Source)
2018-06-13 12:18:59 DEBUG CarrierRefresher:85 - Shutting down the CarrierRefresher.
2018-06-13 12:18:59 DEBUG CarrierRefresher:105 - Error while refreshing bucket config, ignoring.
com.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:551)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.handlerRemoved(AbstractGenericHandler.java:532)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:624)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyDown(DefaultChannelPipeline.java:876)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:842)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:834)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:44)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1284)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:176)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:162)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:819)
at com.couchbase.client.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:717)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Unknown Source)

ismail.iqbal.ap · June 14, 2018, 2:41pm

Can anyone please help me here???

unhuman · August 13, 2019, 6:18pm

We just got this too with version 2.7.4 of the Couchbase Java client:

om.couchbase.client.core.RequestCancelledException: Request cancelled in-flight.
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.handleOutstandingOperations(AbstractGenericHandler.java:686)
	at com.couchbase.client.core.endpoint.AbstractGenericHandler.exceptionCaught(AbstractGenericHandler.java:643)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:259)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:251)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:426)
	at com.couchbase.client.deps.io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:88)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:147)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:233)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:259)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:251)
	at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:259)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:251)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1271)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:280)
	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:259)
	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:909)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 common frames omitted
Wrapped by: com.cvent.archivesibling.exceptions.ArchiveProcessingException: Processing problem (Request cancelled in-flight.) (ERR_INTERNAL)
	at com.cvent.archivesibling.process.AbstractArchiveFramework.processRequest(AbstractArchiveFramework.java:465)
	at com.cvent.archivesibling.resources.ArchiveSiblingResource.backgroundProcessing(ArchiveSiblingResource.java:354)
	at com.cvent.archivesibling.resources.ArchiveSiblingResource.lambda$handleRequest$0(ArchiveSiblingResource.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

pooja_Mamidwar · November 28, 2019, 3:25pm

We are also facing the same issue. Are there any updates on this?