Java Driver 2.3.2 vs 2.3.1: leak (potential)

Hello!
After client driver update (2.3.1 → 2.3.2) caught a log message:

[cb-io-1-1] com.couchbase.client.deps.io.netty.util.ResourceLeakDetector LEAK: ByteBuf.release() was not called before it’s garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option ‘-Dcom.couchbase.client.deps.io.netty.leakDetection.level=advanced’ or call ResourceLeakDetector.setLevel() See Netty.docs: Reference counted objects for more information.

I will re-run with -Dcom.couchbase.client.deps.io.netty.leakDetection.level=advanced later to get additional information.
But anyway there was no such message for 2.3.1 (many runs).

@daschl, @simonbasle
all yours:

[cb-io-1-1] com.couchbase.client.deps.io.netty.util.ResourceLeakDetector LEAK: ByteBuf.release() was not called before it’s garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.#012Recent access records: 2#012#2:#012#011com.couchbase.client.deps.io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByteDesc(AdvancedLeakAwareByteBuf.java:659)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseViewInfo(ViewHandler.java:443)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseQueryResponse(ViewHandler.java:354)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:266)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:70)#012#011com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:229)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)#012#011com.couchbase.client.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:883)#012#011com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:389)#012#011com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:305)#012#011com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)#012#011com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)#012#011java.lang.Thread.run(Thread.java:745)#012#1:#012#011com.couchbase.client.deps.io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:581)#012#011com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:680)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseViewInfo(ViewHandler.java:442)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseQueryResponse(ViewHandler.java:354)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:266)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:70)#012#011com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:229)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com

[cb-io-1-1] com.couchbase.client.deps.io.netty.util.ResourceLeakDetector LEAK: ByteBuf.release() was not called before it’s garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.#012Recent access records: 2#012#2:#012#011com.couchbase.client.deps.io.netty.buffer.AdvancedLeakAwareByteBuf.forEachByteDesc(AdvancedLeakAwareByteBuf.java:659)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseViewInfo(ViewHandler.java:443)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseQueryResponse(ViewHandler.java:354)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:266)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:70)#012#011com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:229)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)#012#011com.couchbase.client.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:883)#012#011com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:389)#012#011com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:305)#012#011com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)#012#011com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)#012#011java.lang.Thread.run(Thread.java:745)#012#1:#012#011com.couchbase.client.deps.io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:581)#012#011com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:680)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseViewInfo(ViewHandler.java:442)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.parseQueryResponse(ViewHandler.java:354)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:266)#012#011com.couchbase.client.core.endpoint.view.ViewHandler.decodeResponse(ViewHandler.java:70)#012#011com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:229)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)#012#011com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)#012#011com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)#012#011com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:345)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)#012#011com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:352)#012#011com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)#012#011com.couchbase.client.deps.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:883)#012#011com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.processReady(E

it seems to be a leak in relation to a view query response’s info() section. Can you confirm you were running view queries? If so, could you also try to capture what the response looks like (either by trying to directly execute the view request in the REST api or by capturing packets)?

Especially I’m interested in seeing if the response doesn’t have any entry before "rows"

(PS: the way the log is copy-pasted makes it pretty hard to read :confused:)

Can you confirm you were running view queries?

Yes

If so, could you also try to capture what the response looks like (either by trying to directly execute the view request in the REST api or by capturing packets)?

I have 3 views, so i need additional logging, to determine which one is the cause. I’ll do it a little bit later.

Especially I’m interested in seeing if the response doesn’t have any entry before “rows”

Mmm… do you mean an “empty response” ? Or “entry” means something like “not-result-containing part of response that was omitted” (should be visible via REST-call)?

(PS: the way the log is copy-pasted makes it pretty hard to read :confused:)

Sorry, that’s direct output from log “as is” (copy-paste).

Mmm… do you mean an “empty response” ? Or “entry” means something like “not-result-containing part of response that was omitted” (should be visible via REST-call)?

The second proposition :slight_smile: View response should usually have the total number of rows then the “rows” entry, which is the array of individual results from the view. From what I could see, if there is nothing before the “rows” entry in the response, an empty ByteBuf could leak.
But I was still unable to reproduce the LEAK message locally though.

Hm, looks like you were right, one of my views does not return “total_rows” on empty data set:

normal:

{“total_rows”:0,“rows”:[
]
}

erroneous:

{“rows”:[
]
}

Is that a server bug / feature ?

@simonbasle,
looks like i got it. The problem is in reduce. If you do have a reduce function defined in view (_count in my case) and use REST-call param “reduce=true”, then result is erroneous. If reduce = false, then result is normal. That’s not reasonable, but explainable because reduce=true implies that there is always 1 row.
So, imho, options are:

  1. Fix this on the server-level (i.e. for reduce=true, total_rows must exists in response and be always = 1)
  2. Make a JavaSDK-workaround (if reduce = true …)
    Second one is the easiest, first one is correct. Make your free choice :wink:

P.S. second way is also more “safe”, because first one will probably require revision of all existing SDK’s

yeah it’s not really a bug I think, but more the way reduce works (where total_rows makes no sense).
It appears to be a 1 byte leak :slight_smile: I’ll go ahead and fix it in the Java SDK :thumbsup:

You gonna fight!
For one byte!
To saaaaaaaave it!

Looking forward for 2.3.3 :slight_smile:

@simonbasle,
With 2.3.3-SNAPSHOT + 1.3.3-SNAPSHOT i don’t see “LEAK: ByteBuf.release() was not called before it’s garbage-collected.” message anymore (200+ runs). Seems like your fix is working :wink: