RxComputationScheduler threads persist despite closing dcp Client

Java SDK
1

Hi,
We create com.couchbase.client.dcp.Client instance to get high sequence numbers for all buckets on a cluster.
So we call below for all buckets in a loop
Client.connect().await(30, TimeUnit.SECONDS)
Client.initializeState(StreamFrom.BEGINNING, StreamTo.NOW).await(30, TimeUnit.SECONDS)
Client.sessionState()
Client.disconnect().await(30, TimeUnit.SECONDS)

Everything finishes as expected however the program does not quit due to some RX threads still being active. Please find jstack output attached.jstack.zip (2.5 KB)

We earlier get list of all buckets with
com.couchbase.client.java.cluster.ClusterManager.getBuckets()
And call com.couchbase.client.java.CouchbaseCluster.disconnect()
and CouchbaseEnvironment .shutdown() after.
Are we missing any cleanup steps ?
DCP SDK 0.19.0
JAVA SDK 2.6.1

2

As a final cleanup step, you can terminate the RX threads by calling rx.schedulers.Schedulers.shutdown().

However, the RX scheduler threads are daemon threads, so they shouldn’t be preventing the program from quitting. Maybe there’s something else going on?

This non-daemon thread looks suspicious:

"nioEventLoopGroup-6-1" #79 prio=10 os_prio=0 tid=0x00007f2ad615d000 nid=0x3eac0 runnable [0x00007f2aa4bad000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0000000781298518> (a com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x00000007812882c0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007812881a8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)
        at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
        at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:745)

@daschl any thoughts?

4

Any more ideas on this ?
We cannot repro this issue, hence cannot confirm if @david.nault 's solution works.

5

I doubt calling rx.schedulers.Schedulers.shutdown() will affect whether or not the app terminates. My money is on that Netty thread being stubborn and not terminating.

Searching the internet for EPollArrayWrapper.epollWait turns up conversations where people recommend upgrading to a new Linux kernel, upgrading the JDK, switching to a different JDK provider, or selecting a different garbage collection algorithm. In other words, lots of voodoo.

The sledgehammer approach would be to disable the native epoll transport. This Stackoverflow post suggests one way to do this is to set the Java system property os.name to anything that doesn’t contain “linux”.

1 Like