IllegalReferenceCountException - Error during IO write phase on couchbase-2.0.3

I updated couchbase sdk version from 2.0.1 to 2.0.3.
and then I met IO error below. Is there any one experienced like this error ?

2015-04-23 18:26:10,034 WARN [com.couchbase.client.core.endpoint.Endpoint] - Error during IO write phase.
com.couchbase.client.deps.io.netty.handler.codec.EncoderException: com.couchbase.client.deps.io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:744)
Caused by: com.couchbase.client.deps.io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
at com.couchbase.client.deps.io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:63)
at com.couchbase.client.core.endpoint.kv.KeyValueHandler.encodeRequest(KeyValueHandler.java:165)
at com.couchbase.client.core.endpoint.kv.KeyValueHandler.encodeRequest(KeyValueHandler.java:79)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.encode(AbstractGenericHandler.java:137)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.encode(AbstractGenericHandler.java:50)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$1.encode(MessageToMessageCodec.java:67)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
… 10 more

Hi @Sam_K,
Sorry this message wasn’t seen earlier :frowning: The latest and greatest is currently 2.1.3 and 2.1.4 will be out shortly.
Did you upgrade since having this problem? If not, you should. The 2.1 branch has fixed a few issues related to IllegalReferenceCountException.
Can you confirm if the issue is still there and if it went away after upgrading?
Thanks

I’m seeing this exact behavior in 2.2.1.

19:13:15.434 [cb-io-1-1] WARN  c.c.client.core.endpoint.Endpoint - Error during IO write phase.
com.couchbase.client.deps.io.netty.handler.codec.EncoderException: com.couchbase.client.deps.io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
    at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) [core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [core-io-1.2.1.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by: com.couchbase.client.deps.io.netty.util.IllegalReferenceCountException: refCnt: 0, increment: 1
    at com.couchbase.client.deps.io.netty.buffer.AbstractReferenceCountedByteBuf.retain(AbstractReferenceCountedByteBuf.java:63) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.core.endpoint.kv.KeyValueHandler.encodeRequest(KeyValueHandler.java:197) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.core.endpoint.kv.KeyValueHandler.encodeRequest(KeyValueHandler.java:93) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.core.endpoint.AbstractGenericHandler.encode(AbstractGenericHandler.java:192) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.core.endpoint.AbstractGenericHandler.encode(AbstractGenericHandler.java:64) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$1.encode(MessageToMessageCodec.java:67) ~[core-io-1.2.1.jar:na]
    at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ~[core-io-1.2.1.jar:na]
    ... 10 common frames omitted

I think basically that the problem occurs when the same document is passed to multiple actions.

In my code, I’ve got something similar to following logic to append a document (or insert if DNE):

document = BinaryDocument.create(key, byteBuffer);
try {
    bucket.append(document);
} catch (DocumentDoesNotExistException dneException) {
    bucket.insert(document);
    // ** ^ BOOM ^ ** //
}

Workaround: If I recreate the ByteBuffer (and then recreate the BinaryDocument) before the insert, I do not get that error.

@unhuman yeah with BinaryDocument you have more responsibilities. We expose the Netty ByteBuf, which can be an off-heap pooled buffer of bytes, so it needs to be managed explicitly to make sure it is returned to the pool and no memory leaks.

Notice that the SDK will release() the buffer whenever it writes it into a message (same as it does with other types of documents actually, this is deeper down the internals).

But since you potentially reuse this buffer in your particular write case, you should increment its reference counter for things to go smoothly:

byteBuffer.retain(); //prepare for potential multi usage (+1 refCnt, refCnt = 2)
try {
   bucket.append(document);
   // refCnt = 2 on success
   byteBuffer.release(); //refCnt = 1
} catch (DocumentDoesNotExistException dneException) {
   // buffer is released on errors, refCnt = 1
   //second usage will also release, but we want to be at refCnt = 1 for the finally block
   byteBuffer.retain(); //refCnt = 2
   bucket.insert(document); //refCnt = 1
} // other uncaught errors will still cause refCnt to be released down to 1
finally {
   //we made sure that at this point refCnt = 1 in any case (success, caught exception, uncaught exception)
   byteBuffer.release(); //refCnt = 0, returned to the pool
}
```

Alternatively:

byteBuffer.retain(); //prepare for potential multi usage (+1 refCnt, refCnt = 2)
try {
bucket.append(document);
// refCnt = 2 on success
byteBuffer.release(2); //success path, free buffer
} catch (DocumentDoesNotExistException dneException) {
// buffer is released (refCnt = 1) on errors, and second usage will free it
bucket.insert(document); //frees buffer
} catch (Exception others) {
// other uncaught errors will still cause refCnt to be released down to 1
// free the buffer by calling release() once more since it’s not used in this path
byteBuffer.release();
}


You have to prepare for the fact before the exception is caught because you cannot `retain()` a buffer that's been `release()` down to a `refCnt()` of 0.
1 Like