Error while connecting containerized couchbase instance using Java-Client

Manikandabalaji · June 4, 2020, 6:47am

I am a naive user to Couchbase. Am working on a requirement where am asked to connect to a Couchbase instance running in OpenShift. We are able to access couchbase ui portal. When I try to connect to the instance using java-client using a springboot application, in logger I get Node connected and after while I get node disconnected message. I am unable to understand the logger information available. Any help will be much appreciated. Thanks in advance.

Code Snippet:

@Bean
public Bucket createCollection() {
	Set<SeedNode> seedNodes = new HashSet<>(Arrays.asList(
			SeedNode.create("<<Node url>>", Optional.of(80), Optional.of(80))));

	ClusterEnvironment env = ClusterEnvironment.builder()
			.timeoutConfig(TimeoutConfig.kvTimeout(Duration.ofSeconds(5)).managementTimeout(Duration.ofSeconds(15))
					.kvDurableTimeout(Duration.ofSeconds(20)))
			.build();

	Cluster cluster = Cluster.connect(seedNodes, ClusterOptions.clusterOptions("<<UserName>>", "<<Password>>")
			.environment(env));
	Bucket bucket = cluster.bucket("<<Bucket_name>>");
	return bucket;
}

Console log:

2020-06-03 20:13:26.824 INFO 18128 — [ cb-events] com.couchbase.core : [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.0.4”,“clientGitHash”:“b0343d25”,“coreVersion”:“2.0.5”,“coreGitHash”:“b0343d25”,“userAgent”:“couchbase-java/3.0.4 (Windows 10 10.0 amd64; Java HotSpot™ 64-Bit Server VM 1.8.0_191-b26)”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:2,“eventLoopGroups”:[“NioEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:30000,“configIdleRedialTimeoutMs”:300000},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:false,“nativeTlsEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:null},“timeoutConfig”:{“kvMs”:2500,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:75000,“viewMs”:75000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:10000,“disconnectMs”:10000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:false,“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“OwnedSupplier”} {“coreId”:“0xd5342be000000001”}
2020-06-03 20:13:26.840 INFO 18128 — [ cb-events] com.couchbase.node : [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xd5342be000000001”,“managerPort”:“80”,“remote”:“dev.insightcouchbase.namoseswd10d.nam.nsroot”}
2020-06-03 20:13:27.199 INFO 18128 — [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService ‘applicationTaskExecutor’
2020-06-03 20:13:27.466 INFO 18128 — [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ‘’
2020-06-03 20:13:27.469 INFO 18128 — [ main] com.knoldus.blogs.Application : Started Application in 4.345 seconds (JVM running for 5.001)
2020-06-03 20:13:36.645 WARN 18128 — [ cb-events] com.couchbase.endpoint : [com.couchbase.endpoint][EndpointConnectionFailedEvent][10s] Connect attempt 1 failed because of TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured) {“circuitBreaker”:“DISABLED”,“coreId”:“0xd5342be000000001”,“remote”:“dev.insightcouchbase.namoseswd10d.nam.nsroot:80”,“type”:“KV”}

java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:288) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:273) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:390) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]

2020-06-03 20:13:36.949 WARN 18128 — [ cb-events] com.couchbase.core : [com.couchbase.core][InitGlobalConfigFailedEvent][10s] Initializing the global config failed: UNKNOWN {“coreId”:“0xd5342be000000001”}

com.couchbase.client.core.error.ConfigException: Caught exception while loading global config.
at com.couchbase.client.core.config.loader.GlobalLoader.lambda$load$3(GlobalLoader.java:70) ~[core-io-2.0.5.jar:na]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:134) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1550) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onError(MonoIgnoreThen.java:306) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:126) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.Operators.error(Operators.java:181) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.Mono.subscribe(Mono.java:3694) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:97) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onError(FluxDoFinally.java:129) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at com.couchbase.client.core.Reactor$SilentMonoCompletionStage.lambda$subscribe$0(Reactor.java:175) ~[core-io-2.0.5.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_191]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_191]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_191]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_191]
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:168) ~[core-io-2.0.5.jar:na]
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157) ~[core-io-2.0.5.jar:na]
at com.couchbase.client.core.deps.ioty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[core-io-2.0.5.jar:na]
at com.couchbase.client.core.deps.ioty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[core-io-2.0.5.jar:na]
at com.couchbase.client.core.deps.ioty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[core-io-2.0.5.jar:na]
at com.couchbase.client.core.deps.ioty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[core-io-2.0.5.jar:na]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
Caused by: com.couchbase.client.core.error.UnambiguousTimeoutException: CarrierGlobalConfigRequest
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:163) ~[core-io-2.0.5.jar:na]
… 6 common frames omitted

2020-06-03 20:13:36.950 WARN 18128 — [ cb-events] com.couchbase.endpoint : [com.couchbase.endpoint][EndpointConnectionFailedEvent][10s] Connect attempt 1 failed because of TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured) {“bucket”:“insight_auth”,“circuitBreaker”:“DISABLED”,“coreId”:“0xd5342be000000001”,“remote”:“dev.insightcouchbase.namoseswd10d.nam.nsroot:80”,“type”:“KV”}

java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 10000ms in ‘source(MonoDefer)’ (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:288) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:273) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:390) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27) ~[reactor-core-3.2.6.RELEASE.jar:3.2.6.RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_191]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]

2020-06-03 20:13:37.152 INFO 18128 — [ cb-events] com.couchbase.node : [com.couchbase.node][NodeDisconnectedEvent][2854us] Node disconnected {“coreId”:“0xd5342be000000001”,“managerPort”:“80”,“remote”:“dev.insightcouchbase.namoseswd10d.nam.nsroot”}

mreiche · June 5, 2020, 2:12am

80 is not the normal/typical port number for kv, it’s usually 11210, but perhaps you have it mapped to 80.

david.nault · June 5, 2020, 2:17am

Hi Manikandabalaji,

Is your Spring application running on OpenShift too, or are you connecting from outside OpenShift?

If you’re connecting from outside, you may need to configure your Couchbase Server nodes to advertise alternate addresses.

https://docs.couchbase.com/server/current/learn/clusters-and-availability/connectivity.html#alternate-addresses

Then, to make absolutely sure your client is using the right set of addresses, you can pin the network resolution to EXTERNAL, like this:

ClusterEnvironment env = ClusterEnvironment.builder()
   .ioConfig(IoConfig.networkResolution(NetworkResolution.EXTERNAL))
   .timeoutConfig(...)
   .build();

(The default which is NetworkResolution.AUTO should work too, but it isn’t infallible.)

EDIT: Upon preview, mreiche is absolutely correct. Those port numbers are suspicious.

Thanks,
David