There is a slight subtility with shutdown though: CouchbaseEnvironment.shutdown() returns an Observable<Boolean> and as such it needs to be subscribed. So your code should be:
//shutdown
//bucket.close(); //this will be called when disconnecting the cluster
cluster.disconnect();
//note: as soon as the env was created by you, you must call shutdown() on it
//here we trigger subscription and wait for termination by blocking on the Observable
couchbaseEnvironment.shutdown().toBlocking().single();
Schedulers.shutdown(); //reordered, last
I think that with this modification, things should be far better.
Improvements on that front have been submitted to master.
prefer using upcoming 2.2.1 release (snapshot can be built from current master) and upgrade to RxJava 1.0.15 as soon as it comes out in order to be able to call rx.Schedulers.shutdown()
don’t forget to call toBlocking().single() (or at least subscribe()) after an environment.shutdown()
Note: The improvements have been partially backported to release11 branch for inclusion in the upcoming 2.1.5 release, but improvements for RxJava and Netty threads couldn’t be backported, so it’s mainly clarity of code, integration tests and logs that have been backported.
Really god! We’re in the right way.
I’ve set the last cb dependencies based on master branch and the last memory leak is this threadDeathWatcher:
13:42:04.218 [cb-io-1-4] DEBUG c.c.c.d.i.n.buffer.PoolThreadCache - Freed 21 thread-local buffer(s) from thread: cb-io-1-4
13:42:04.218 [cb-io-1-3] DEBUG c.c.c.d.i.n.buffer.PoolThreadCache - Freed 10 thread-local buffer(s) from thread: cb-io-1-3
13:42:04.218 [cb-io-1-2] DEBUG c.c.c.d.i.n.buffer.PoolThreadCache - Freed 16 thread-local buffer(s) from thread: cb-io-1-2
13:42:04.219 [cb-io-1-1] DEBUG c.c.c.d.i.n.buffer.PoolThreadCache - Freed 13 thread-local buffer(s) from thread: cb-io-1-1 SEVERE: The web application [/mauricio] appears to have started a thread named [threadDeathWatcher-4-1] but has failed to stop it. This is very likely to create a memory leak.
Note: The improvements have been partially backported to release11 branch for inclusion in the upcoming 2.1.5 release, but improvements for RxJava and Netty threads couldn’t be backported, so it’s mainly clarity of code, integration tests and logs that have been backported.
Any specific reason of why Netty threads couldn’t be backported?
Just for knowledge, after build java client from master branch and use the 2.2.1 release, I started to get this error during some operations:
Exception in thread "cb-computations-4" java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread.
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:62)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: com.couchbase.client.core.message.kv.UpsertResponse.mutationToken()Lcom/couchbase/client/core/message/kv/MutationToken;
at com.couchbase.client.java.CouchbaseAsyncBucket$16.call(CouchbaseAsyncBucket.java:501)
at com.couchbase.client.java.CouchbaseAsyncBucket$16.call(CouchbaseAsyncBucket.java:493)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
at rx.observers.Subscribers$5.onNext(Subscribers.java:234)
at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:222)
at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:101)
at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:199)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
… 7 more
@mcarvalho it looks like there is something wrong with your classpath, mutation tokens have been added in 2.2.0 / 1.2.0 and for some reason they are not found.
mmh did you also build core-io project (couchbase-jvm-core) from master? and if so, can you make sure the core-io version and the corresponding dependency match in both projects pom.xml?
Hi,
Facing issue with leaking rx threads, we have managed to eliminate the issue by running 2.1.4 with RXJava snapshot(1.0.15). and call rx.Schedulers.shutdown().
Is there any open ticket within couchbase regarding this issue, eg. updating the RxJava version as soon as it’s available?
yes there was a ticket for the threads leaking, see post #31 above…
we’ll upgrade to RxJava 1.0.15 (or above) in the 2.2.x bugfix release that follows the RxJava release, hopefully next one.
yeah there was also a kind of “parent” change https://issues.couchbase.com/browse/JCBC-773
the 251 change is in the core project, so it has core-io numbering scheme… it also only targets SDK threads…
note that you don’t necessary have to wait for us to bump the minimum dependency of the SDK to RxJava 1.0.15 once it is out (this may take some time), you can use it explicitly as soon as it is officially out.
@simonbasle@daschl Thank for helping until here, I really appreciate it.
After these advices, lib updates and shutdown changes, I’m able to not see any Thread lock error anymore although after disconnect from cb cluster and shutdown the env + schedulers, the cb pooling system reconnect in a blink and after that maintain pooling active even after the application shutdown at all, in some way it still there pooling and checking the buckets, creating a memory leak .
Just for curious, I know this pooling system is something new included in the last version. Is possible to disable it before shutdown or for dev/test environments?
We are running in a cluster with 3 nodes, using 3 buckets each. it looks like below during application execution:
11:48:33.623 [cb-core-3-1] DEBUG c.c.c.c.config.ConfigurationProvider - Received signal for outdated configuration.
11:48:33.623 [cb-core-3-1] DEBUG c.c.c.c.config.ConfigurationProvider - Received signal for outdated configuration.
11:48:33.638 [cb-computations-3] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank-revisions config proposed.
11:48:33.639 [cb-computations-4] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket ordering config proposed.
11:48:33.646 [cb-computations-1] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket ordering config proposed.
11:48:33.649 [cb-computations-3] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank-revisions"
11:48:33.650 [cb-computations-3] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank-revisions config proposed.
11:48:33.650 [cb-computations-1] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "ordering"
11:48:33.651 [cb-computations-3] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank-revisions"
11:48:33.653 [cb-computations-2] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket ordering config proposed.
11:48:33.657 [cb-computations-3] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank config proposed.
11:48:33.657 [cb-computations-4] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "ordering"
11:48:33.657 [cb-computations-1] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank config proposed.
11:48:33.660 [cb-computations-4] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank-revisions config proposed.
11:48:33.661 [cb-computations-2] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "ordering"
11:48:33.661 [cb-computations-2] DEBUG c.c.c.c.config.ConfigurationProvider - New Bucket databank config proposed.
11:48:33.662 [cb-computations-1] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank"
11:48:33.662 [cb-computations-4] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank-revisions"
11:48:33.662 [cb-computations-3] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank"
11:48:33.663 [cb-computations-2] DEBUG c.c.c.c.c.refresher.CarrierRefresher - Completed refreshing config for bucket "databank"
the connection polling (JVMCBC-257) has been fixed and will be part of the upcoming 2.2.2 release, expected by next week at most.
also great news, RxJava 1.0.15 is officially out
(we’ve bumped dependency to 1.0.15 in SDK 2.2.2 by the way)
Hello @simonbasle. I’m back
After update to RxJava 1.0.15 and couchbase 2.2.2 and core-io 1.2.2 I’m getting this memory leak
SEVERE: The web application [/mauricio] appears to have started a thread named [initialSeedUniquifierGenerator] but has failed to stop it. This is very likely to create a memory leak.
I had noticed that env.shutdown().toBlocking().single(); //is marked as deprecated.
Do you think it is related?
By the way, what should I use instead of it?
My environment is the same as described above with the libs updated.
The deprecated warning is fine, it is there to avoid someone doing just env.shutdown(), but in your case you are actually blocking on the call so it’s all fine. We’ll migrate shutdown to be blocking (not returning an Observable) so we have introduced shutdownAsync for the non-blocking semantics. Up to you if you want to use that or fix the code when it becomes blocking later on (probably not before 2.3).
The leak warning seems to be coming from Netty (and specifically RandomThreadLocal) . Not sure of the actual workaround there You do not use Netty do you? (just to check this is due to the SDK’s Netty)
@simonbasle sorry about the last one. I had updated the RXJava from snapshot to current 1.0.15 and it is working now without memory leaks logs. Unfortunately looks like the refresher continue working after shutdown the application it reconnect to our buckets in a blink of eyes.
prefer using upcoming 
don’t forget to call 
.
You do not use Netty do you? (just to check this is due to the SDK’s Netty)