Hi,
We create com.couchbase.client.dcp.Client instance to get high sequence numbers for all buckets on a cluster.
So we call below for all buckets in a loop Client.connect().await(30, TimeUnit.SECONDS) Client.initializeState(StreamFrom.BEGINNING, StreamTo.NOW).await(30, TimeUnit.SECONDS) Client.sessionState() Client.disconnect().await(30, TimeUnit.SECONDS)
Everything finishes as expected however the program does not quit due to some RX threads still being active. Please find jstack output attached.jstack.zip (2.5 KB)
We earlier get list of all buckets with com.couchbase.client.java.cluster.ClusterManager.getBuckets()
And call com.couchbase.client.java.CouchbaseCluster.disconnect()
and CouchbaseEnvironment .shutdown() after.
Are we missing any cleanup steps ?
DCP SDK 0.19.0
JAVA SDK 2.6.1
However, the RX scheduler threads are daemon threads, so they shouldn’t be preventing the program from quitting. Maybe there’s something else going on?
This non-daemon thread looks suspicious:
"nioEventLoopGroup-6-1" #79 prio=10 os_prio=0 tid=0x00007f2ad615d000 nid=0x3eac0 runnable [0x00007f2aa4bad000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000781298518> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000007812882c0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007812881a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:745)
I doubt calling rx.schedulers.Schedulers.shutdown() will affect whether or not the app terminates. My money is on that Netty thread being stubborn and not terminating.
Searching the internet for EPollArrayWrapper.epollWait turns up conversations where people recommend upgrading to a new Linux kernel, upgrading the JDK, switching to a different JDK provider, or selecting a different garbage collection algorithm. In other words, lots of voodoo.
The sledgehammer approach would be to disable the native epoll transport. This Stackoverflow post suggests one way to do this is to set the Java system property os.name to anything that doesn’t contain “linux”.