Not able to connect the more than 16 buckets on startup

In my application I am loading the buckets in application startup and its able to load up to 16 buckets but when it’s tried to connect 17th bucket getting below error.

Exception in thread "main" com.couchbase.client.core.error.UnambiguousTimeoutException: WaitUntilReady timed out {"bucket":"b12","checkedServices":[],"currentState":"ONLINE","desiredState":"ONLINE","services":{"kv":[{"last_activity_us":100650208,"namespace":"b1","state":"connected","id":"0xafc8437b","remote":"172.21.144.1:11210","local":"172.21.144.1:56641"},{"last_activity_us":100427826,"namespace":"b2","state":"connected","id":"0x458c10c6","remote":"172.21.144.1:11210","local":"172.21.144.1:56650"},{"last_activity_us":100373976,"namespace":"b3","state":"connected","id":"0x7e31c098","remote":"172.21.144.1:11210","local":"172.21.144.1:56652"},{"last_activity_us":100067368,"namespace":"b4","state":"connected","id":"0x472d1b42","remote":"172.21.144.1:11210","local":"172.21.144.1:56666"},{"last_activity_us":100517605,"namespace":"b5","state":"connected","id":"0x4f1a40ba","remote":"172.21.144.1:11210","local":"172.21.144.1:56646"},{"last_activity_us":100567578,"namespace":"b6","state":"connected","id":"0x8cd591af","remote":"172.21.144.1:11210","local":"172.21.144.1:56644"},{"last_activity_us":100476652,"namespace":"b7","state":"connected","id":"0x26d428a8","remote":"172.21.144.1:11210","local":"172.21.144.1:56648"},{"last_activity_us":100323750,"namespace":"b8","state":"connected","id":"0x757140a3","remote":"172.21.144.1:11210","local":"172.21.144.1:56655"},{"last_activity_us":100015132,"namespace":"b9","state":"connected","id":"0xbe4812f7","remote":"172.21.144.1:11210","local":"172.21.144.1:56669"},{"last_activity_us":100146117,"namespace":"b10","state":"connected","id":"0xe411a3a4","remote":"172.21.144.1:11210","local":"172.21.144.1:56662"},{"last_activity_us":100612407,"namespace":"b11","state":"connected","id":"0xb72cb85d","remote":"172.21.144.1:11210","local":"172.21.144.1:56642"},{"last_activity_us":19946869,"namespace":"b12","state":"connected","id":"0xefc06535","remote":"172.21.144.1:11210","local":"172.21.144.1:56670"},{"last_activity_us":1090447,"state":"connected","id":"0x17da3bdc","remote":"172.21.144.1:11210","local":"172.21.144.1:56632"},{"last_activity_us":100197351,"namespace":"b13","state":"connected","id":"0x84054f0b","remote":"172.21.144.1:11210","local":"172.21.144.1:56661"},{"last_activity_us":100107606,"namespace":"b14","state":"connected","id":"0x34b9a705","remote":"172.21.144.1:11210","local":"172.21.144.1:56664"},{"last_activity_us":100705915,"namespace":"b15","state":"connected","id":"0xf87ffed0","remote":"172.21.144.1:11210","local":"172.21.144.1:56633"},{"last_activity_us":100285015,"namespace":"b16","state":"connected","id":"0x6aae88f5","remote":"172.21.144.1:11210","local":"172.21.144.1:56656"},{"last_activity_us":100236539,"namespace":"b17","state":"connected","id":"0x21b368c8","remote":"172.21.144.1:11210","local":"172.21.144.1:56658"}],"mgmt":[{"last_activity_us":100717406,"state":"connected","id":"0x179ecc1d","remote":"172.21.144.1:8091","local":"172.21.144.1:56634"},{"state":"connected","id":"0x1b558098","remote":"172.21.144.1:8091","local":"172.21.144.1:56635"},{"last_activity_us":100016089,"state":"connected","id":"0x95a45d93","remote":"172.21.144.1:8091","local":"172.21.144.1:56638"},{"state":"connected","id":"0x9b8e9d6c","remote":"172.21.144.1:8091","local":"172.21.144.1:56643"},{"state":"connected","id":"0xf0671f79","remote":"172.21.144.1:8091","local":"172.21.144.1:56645"},{"state":"connected","id":"0xea93ce4e","remote":"172.21.144.1:8091","local":"172.21.144.1:56647"},{"state":"connected","id":"0x8a188b23","remote":"172.21.144.1:8091","local":"172.21.144.1:56649"},{"state":"connected","id":"0x99be0085","remote":"172.21.144.1:8091","local":"172.21.144.1:56651"},{"state":"connected","id":"0xe357e6d1","remote":"172.21.144.1:8091","local":"172.21.144.1:56653"},{"state":"connected","id":"0x3bcb177e","remote":"172.21.144.1:8091","local":"172.21.144.1:56654"},{"state":"connected","id":"0x16b2c052","remote":"172.21.144.1:8091","local":"172.21.144.1:56657"},{"state":"connected","id":"0xf5e59ce3","remote":"172.21.144.1:8091","local":"172.21.144.1:56659"},{"state":"connected","id":"0xac65048f","remote":"172.21.144.1:8091","local":"172.21.144.1:56660"},{"state":"connected","id":"0x0a55a4e6","remote":"172.21.144.1:8091","local":"172.21.144.1:56663"},{"state":"connected","id":"0x0edd3712","remote":"172.21.144.1:8091","local":"172.21.144.1:56665"},{"state":"connected","id":"0x73aca808","remote":"172.21.144.1:8091","local":"172.21.144.1:56668"}]},"state":{"current_stage":"CONFIG_LOAD","current_stage_since_ms":100004,"timings_ms":{},"total_ms":100004},"timeoutMs":100000}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Bucket.waitUntilReady(Bucket.java:229)
	at poc.BucketInitialization.main(BucketInitialization.java:56)
	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
		at com.couchbase.client.core.diagnostics.WaitUntilReadyHelper.lambda$waitUntilReady$9(WaitUntilReadyHelper.java:162)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
		at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:748)

some times getting below error instead above

Exception in thread "main" com.couchbase.client.core.error.UnambiguousTimeoutException: GenericManagerRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xe55e9bd300000001","idempotent":true,"lastDispatchedTo":"172.21.144.1","reason":"TIMEOUT","requestId":116,"requestType":"GenericManagerRequest","retried":156,"retryReasons":["ENDPOINT_NOT_AVAILABLE"],"timeoutMs":75000}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Bucket.waitUntilReady(Bucket.java:229)
	at poc.BucketInitialization.main(BucketInitialization.java:56)
	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
		at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:184)
		at com.couchbase.client.core.msg.Request.cancel(Request.java:70)
		at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
		at com.couchbase.client.core.deps.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
		at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
		at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)

Below is my code

Cluster cluster = null;
        Bucket bucket = null;
        Map<String, Collection> collectionMap = new HashMap<>();

        Set<SeedNode> seedNodes = new HashSet<>(Collections.singletonList(
                SeedNode.create("172.21.144.1")
        ));
        //KVTimeout: The Key/Value default timeout is used on operations which are performed on a specific key, Default: 2.5s
        ClusterEnvironment env = ClusterEnvironment.builder()
                .timeoutConfig(TimeoutConfig.kvTimeout(Duration.ofMillis(5000)).connectTimeout(Duration.ofMillis(20000)))
                .build();
        cluster = Cluster.connect(seedNodes, ClusterOptions.clusterOptions(username, password).environment(env));
        List<String> bucketList = bucketList();
        long totalTime = 0L;

        for (String bucketName : bucketList) {
            long startTime = System.currentTimeMillis();
            bucket = cluster.bucket(bucketName);
            // get a collection reference
            Collection collection = bucket.defaultCollection();
            bucket.waitUntilReady(Duration.ofSeconds(100));
            collectionMap.put(bucketName, collection);
            long timeTaken = System.currentTimeMillis() - startTime;
            totalTime += timeTaken;
            System.out.println("## Time taken to initialized the bucket:" + bucketName + " = " + timeTaken);
            //Thread.sleep(1000);
        }
        System.out.println("### Average time:" + (totalTime/bucketList.size()) + " for no. of bucket:" + bucketList.size());

I have also tried for increase time for waitUntilReady however not working for me.

Using
Couchbase Java SDK version 3.3.4
Couchbase community server version couchbase-server-community-6.6.0-7909

Can you show all the output from your test? Is it possible that the 17th bucket does not exist?

Running against enterprise server, I have no problem connecting to 20 buckets.

2023-01-11 11:34:09,458  INFO                       com.couchbase.core: 442 - [com.couchbase.core][DnsSrvLookupFailedEvent][68ms] DNS SRV lookup failed (name not found). This is expected if the there is no DNS SRV record associated with the hostname in the connection string. Will now try to bootstrap directly from the given hostname. To suppress this message, specify an IP address instead of a hostname (for example: 127.0.0.1 instead of localhost), specify more than one hostname, or set the `io.enableDnsSrv` client setting to false.
2023-01-11 11:34:09,766  INFO                       com.couchbase.core: 442 - [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.4.1","clientGitHash":"${buildNumber}","coreVersion":"2.4.1","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.4.1 (Mac OS X 10.15.7 x86_64; Java HotSpot(TM) 64-Bit Server VM 17.0.1+12-LTS-39)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":6,"eventLoopGroups":["KQueueEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","backupCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"trustCertificates":null,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":2500,"kvDurableMs":10000,"kvScanMs":75000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000,"eventingMs":75000,"backupMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"consoleLoggerFormatter":"DefaultLoggerFormatter","disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000,"transactionsThresholdMs":5000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0,"scheduler":"ParallelScheduler","schedulerThreadCount":12,"transactionsConfig":{"durabilityLevel":"MAJORITY","timeoutMs":15000,"cleanupConfig":{"runLostAttemptsCleanupThread":true,"runRegularAttemptsCleanupThread":true,"cleanupWindowMs":60000,"cleanupSet":""},"numAtrs":1024,"metadataCollection":"none","scanConsistency":"none"}} {"connectionString":"localhost","coreId":"0x1a2de56500000001","numCoreInstances":1,"seedNodes":[{"address":"localhost"}]}
2023-01-11 11:34:09,767 DEBUG m.couchbase.transactions.cleanup.regular: 427 - [com.couchbase.transactions.cleanup.regular][LogEvent] Starting background cleanup thread to find transactions from this client
2023-01-11 11:34:09,767  INFO               com.couchbase.transactions: 442 - [com.couchbase.transactions][TransactionsStartedEvent] Transactions successfully initialised, regular cleanup enabled=true, lost cleanup enabled=true
2023-01-11 11:34:09,770  INFO                       com.couchbase.node: 442 - [com.couchbase.node][NodeCreatedEvent] Node created {"coreId":"0x1a2de56500000001","managerPort":"8091","remote":"localhost"}
2023-01-11 11:34:10,076  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][338ms] Opened bucket "0" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:10,076  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][358ms] Opened bucket "my_bucket" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:0 = 703
## Time taken to initialized the bucket:1 = 79
2023-01-11 11:34:10,490  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][38ms] Opened bucket "1" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:2 = 76
2023-01-11 11:34:10,595  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][42ms] Opened bucket "2" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:3 = 117
2023-01-11 11:34:10,701  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][73ms] Opened bucket "3" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:10,701  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][28ms] Opened bucket "4" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:4 = 63
## Time taken to initialized the bucket:5 = 63
2023-01-11 11:34:10,807  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][26ms] Opened bucket "5" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:10,807  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][26ms] Opened bucket "6" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:6 = 65
## Time taken to initialized the bucket:7 = 64
2023-01-11 11:34:10,912  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][30ms] Opened bucket "7" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:8 = 64
2023-01-11 11:34:11,017  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][26ms] Opened bucket "8" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:11,017  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][28ms] Opened bucket "9" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:9 = 64
## Time taken to initialized the bucket:a = 62
2023-01-11 11:34:11,123  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][32ms] Opened bucket "a" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:11,123  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][26ms] Opened bucket "b" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:b = 64
## Time taken to initialized the bucket:c = 63
2023-01-11 11:34:11,225  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][25ms] Opened bucket "c" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:d = 61
2023-01-11 11:34:11,329  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][30ms] Opened bucket "d" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:11,329  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][30ms] Opened bucket "e" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:e = 62
## Time taken to initialized the bucket:g = 61
2023-01-11 11:34:11,434  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][25ms] Opened bucket "g" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:11,435  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][25ms] Opened bucket "h" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:h = 66
## Time taken to initialized the bucket:i = 62
2023-01-11 11:34:11,540  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][28ms] Opened bucket "i" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:j = 61
2023-01-11 11:34:11,643  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][30ms] Opened bucket "j" {"coreId":"0x1a2de56500000001"}
2023-01-11 11:34:11,643  INFO                       com.couchbase.core: 442 - [com.couchbase.core][BucketOpenedEvent][26ms] Opened bucket "k" {"coreId":"0x1a2de56500000001"}
## Time taken to initialized the bucket:k = 64
### Average time:99 for no. of bucket:20

Buckets are already there and below is all output of the program and buckets screenshot

"C:\Program Files\Java\jdk1.8.0_221\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2021.1.3\lib\idea_rt.jar=49998:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2021.1.3\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_221\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_221\jre\lib\rt.jar;D:\Tech\Java\CouchbasePOCMaven\target\classes;D:\Projects\.m2\repository\com\couchbase\client\java-client\3.3.4\java-client-3.3.4.jar;D:\Projects\.m2\repository\com\couchbase\client\core-io\2.3.4\core-io-2.3.4.jar;D:\Projects\.m2\repository\io\projectreactor\reactor-core\3.4.22\reactor-core-3.4.22.jar;D:\Projects\.m2\repository\org\reactivestreams\reactive-streams\1.0.4\reactive-streams-1.0.4.jar;D:\Projects\.m2\repository\com\couchbase\client\couchbase-client\1.4.12\couchbase-client-1.4.12.jar;D:\Projects\.m2\repository\io\netty\netty\3.5.5.Final\netty-3.5.5.Final.jar;D:\Projects\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;D:\Projects\.m2\repository\stax\stax-api\1.0.1\stax-api-1.0.1.jar;D:\Projects\.m2\repository\commons-codec\commons-codec\1.5\commons-codec-1.5.jar;D:\Projects\.m2\repository\net\spy\spymemcached\2.12.1\spymemcached-2.12.1.jar;D:\Projects\.m2\repository\org\apache\httpcomponents\httpcore\4.3\httpcore-4.3.jar;D:\Projects\.m2\repository\org\apache\httpcomponents\httpcore-nio\4.3\httpcore-nio-4.3.jar" poc.BucketInitialization
Jan 12, 2023 12:28:23 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.3.4","clientGitHash":"${buildNumber}","coreVersion":"2.3.4","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.3.4 (Windows 10 10.0 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_221-b11)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["NioEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","backupCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"trustCertificates":null,"trustManagerFactory":null,"ciphers":[]},"timeoutConfig":{"kvMs":5000,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":20000,"disconnectMs":10000,"eventingMs":75000,"backupMs":75000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"consoleLogLevel":{"name":"INFO","resourceBundleName":"sun.util.logging.resources.logging","localizedName":"INFO"},"consoleLoggerFormatter":"DefaultLoggerFormatter","disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"enabled":true},"thresholdLoggingTracerConfig":{"enabled":true,"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"loggingMeterConfig":{"enabled":true,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0,"scheduler":"ParallelScheduler","schedulerThreadCount":8,"transactionsConfig":{"durabilityLevel":"MAJORITY","timeoutMs":15000,"cleanupConfig":{"runLostAttemptsCleanupThread":true,"runRegularAttemptsCleanupThread":true,"cleanupWindowMs":60000,"cleanupSet":""},"numAtrs":1024,"metadataCollection":"none","scanConsistency":"none"}} {"coreId":"0x8a683bca00000001","numCoreInstances":1,"seedNodes":[{"address":"172.21.144.1"}]}
Jan 12, 2023 12:28:23 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.transactions][TransactionsStartedEvent] Transactions successfully initialised, regular cleanup enabled=true, lost cleanup enabled=true
Jan 12, 2023 12:28:23 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x8a683bca00000001","managerPort":"8091","remote":"172.21.144.1"}
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][1537ms] Opened bucket "ActionPluginCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:ActionPluginCache = 1762
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][78ms] Opened bucket "amsSolrData" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:amsSolrData = 144
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][56ms] Opened bucket "asiteStorageRepositoryCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:asiteStorageRepositoryCache = 99
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][54ms] Opened bucket "AsiteWorksCustomCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:AsiteWorksCustomCache = 95
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][43ms] Opened bucket "catalogDetails" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:catalogDetails = 92
## Time taken to initialized the bucket:customConfigCache = 90
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][57ms] Opened bucket "customConfigCache" {"coreId":"0x8a683bca00000001"}
Jan 12, 2023 12:28:27 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][67ms] Opened bucket "docStatusCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:docStatusCache = 125
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][76ms] Opened bucket "emailFormContentCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:emailFormContentCache = 183
## Time taken to initialized the bucket:folderAdminCache = 189
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][112ms] Opened bucket "folderAdminCache" {"coreId":"0x8a683bca00000001"}
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][74ms] Opened bucket "forms" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:forms = 143
## Time taken to initialized the bucket:MasterDataCache = 151
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][89ms] Opened bucket "MasterDataCache" {"coreId":"0x8a683bca00000001"}
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][69ms] Opened bucket "MetaDataCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:MetaDataCache = 125
Jan 12, 2023 12:28:28 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][62ms] Opened bucket "modelData" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:modelData = 102
## Time taken to initialized the bucket:poiCache = 89
Jan 12, 2023 12:28:29 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][39ms] Opened bucket "poiCache" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:progressData = 91
Jan 12, 2023 12:28:29 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][54ms] Opened bucket "progressData" {"coreId":"0x8a683bca00000001"}
## Time taken to initialized the bucket:projectDC = 102
Jan 12, 2023 12:28:29 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info
INFO: [com.couchbase.core][BucketOpenedEvent][59ms] Opened bucket "projectDC" {"coreId":"0x8a683bca00000001"}
Exception in thread "main" com.couchbase.client.core.error.UnambiguousTimeoutException: WaitUntilReady timed out {"bucket":"revisionDetails","checkedServices":[],"currentState":"ONLINE","desiredState":"ONLINE","services":{"kv":[{"last_activity_us":101697015,"namespace":"amsSolrData","state":"connected","id":"0xff6787b3","remote":"172.21.144.1:11210","local":"172.21.144.1:50099"},{"last_activity_us":101195865,"namespace":"docStatusCache","state":"connected","id":"0x206c7bad","remote":"172.21.144.1:11210","local":"172.21.144.1:50104"},{"last_activity_us":101009927,"namespace":"emailFormContentCache","state":"connected","id":"0x8c29c2f7","remote":"172.21.144.1:11210","local":"172.21.144.1:50106"},{"last_activity_us":100119371,"namespace":"progressData","state":"connected","id":"0x02a6f23a","remote":"172.21.144.1:11210","local":"172.21.144.1:50120"},{"last_activity_us":101409460,"namespace":"catalogDetails","state":"connected","id":"0x5afd1e20","remote":"172.21.144.1:11210","local":"172.21.144.1:50102"},{"last_activity_us":101499365,"namespace":"AsiteWorksCustomCache","state":"connected","id":"0x0b95ff9c","remote":"172.21.144.1:11210","local":"172.21.144.1:50101"},{"last_activity_us":101317967,"namespace":"customConfigCache","state":"connected","id":"0x9ded3fdc","remote":"172.21.144.1:11210","local":"172.21.144.1:50103"},{"last_activity_us":100819984,"namespace":"folderAdminCache","state":"connected","id":"0x9c0181fc","remote":"172.21.144.1:11210","local":"172.21.144.1:50112"},{"last_activity_us":100017156,"namespace":"projectDC","state":"connected","id":"0x0b1ba22a","remote":"172.21.144.1:11210","local":"172.21.144.1:50121"},{"last_activity_us":100299162,"namespace":"modelData","state":"connected","id":"0x00c34d1a","remote":"172.21.144.1:11210","local":"172.21.144.1:50117"},{"last_activity_us":101598982,"namespace":"asiteStorageRepositoryCache","state":"connected","id":"0x5c259b81","remote":"172.21.144.1:11210","local":"172.21.144.1:50100"},{"last_activity_us":19917389,"namespace":"revisionDetails","state":"connected","id":"0x0bf9f7a3","remote":"172.21.144.1:11210","local":"172.21.144.1:50122"},{"last_activity_us":2456097,"state":"connected","id":"0xf3db9b35","remote":"172.21.144.1:11210","local":"172.21.144.1:50086"},{"last_activity_us":100415629,"namespace":"MetaDataCache","state":"connected","id":"0x00ef6537","remote":"172.21.144.1:11210","local":"172.21.144.1:50116"},{"last_activity_us":100209688,"namespace":"poiCache","state":"connected","id":"0x9c9a80e1","remote":"172.21.144.1:11210","local":"172.21.144.1:50118"},{"last_activity_us":101865284,"namespace":"ActionPluginCache","state":"connected","id":"0x65a6a264","remote":"172.21.144.1:11210","local":"172.21.144.1:50087"},{"last_activity_us":100677792,"namespace":"forms","state":"connected","id":"0x6c6b875c","remote":"172.21.144.1:11210","local":"172.21.144.1:50113"},{"last_activity_us":100529767,"namespace":"MasterDataCache","state":"connected","id":"0x168c819a","remote":"172.21.144.1:11210","local":"172.21.144.1:50115"}],"mgmt":[{"last_activity_us":100306944,"state":"connected","id":"0x8ef53e57","remote":"172.21.144.1:8091","local":"172.21.144.1:50088"},{"state":"connected","id":"0x8859f757","remote":"172.21.144.1:8091","local":"172.21.144.1:50089"},{"last_activity_us":100025721,"state":"connected","id":"0x3a0ef3b2","remote":"172.21.144.1:8091","local":"172.21.144.1:50090"},{"last_activity_us":101917906,"state":"connected","id":"0x9ac378e3","remote":"172.21.144.1:8091","local":"172.21.144.1:50091"},{"last_activity_us":101715949,"state":"connected","id":"0x80e49689","remote":"172.21.144.1:8091","local":"172.21.144.1:50092"},{"last_activity_us":101587454,"state":"connected","id":"0x5d1eca1b","remote":"172.21.144.1:8091","local":"172.21.144.1:50093"},{"last_activity_us":101498359,"state":"connected","id":"0x0bda367f","remote":"172.21.144.1:8091","local":"172.21.144.1:50094"},{"last_activity_us":101414007,"state":"connected","id":"0xa28e427c","remote":"172.21.144.1:8091","local":"172.21.144.1:50095"},{"state":"connected","id":"0x0ffad496","remote":"172.21.144.1:8091","local":"172.21.144.1:50096"},{"state":"connected","id":"0xb12c63da","remote":"172.21.144.1:8091","local":"172.21.144.1:50105"},{"last_activity_us":101016736,"state":"connected","id":"0xd99a974d","remote":"172.21.144.1:8091","local":"172.21.144.1:50107"},{"last_activity_us":100817557,"state":"connected","id":"0xf254bfe5","remote":"172.21.144.1:8091","local":"172.21.144.1:50108"},{"last_activity_us":100675965,"state":"connected","id":"0x72bc9d0f","remote":"172.21.144.1:8091","local":"172.21.144.1:50109"},{"last_activity_us":100540426,"state":"connected","id":"0x98e99713","remote":"172.21.144.1:8091","local":"172.21.144.1:50110"},{"last_activity_us":100416340,"state":"connected","id":"0xa3eb0d82","remote":"172.21.144.1:8091","local":"172.21.144.1:50111"},{"last_activity_us":100211191,"state":"connected","id":"0xebdabef9","remote":"172.21.144.1:8091","local":"172.21.144.1:50119"}]},"state":{"current_stage":"CONFIG_LOAD","current_stage_since_ms":100009,"timings_ms":{},"total_ms":100009},"timeoutMs":100000}
	at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
	at com.couchbase.client.java.Bucket.waitUntilReady(Bucket.java:229)
	at poc.BucketInitialization.main(BucketInitialization.java:58)
	Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
		at com.couchbase.client.core.diagnostics.WaitUntilReadyHelper.lambda$waitUntilReady$9(WaitUntilReadyHelper.java:162)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4397)
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:301)
		at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
		at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
		at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:748)

Process finished with exit code 1

From above output the bucket “revisionDetails” is already there however not able to connect it.(even tried the increasnig the waitUntilReady time to 300 seconds but still same error WaitUntilReady timed out)
Here we can see all buckets are up

Instead of the waitUntilReady, you could try a bucket.get(“dummyid”) and catch the document not found exception. That should give a better diagnostic than waitUntilReady.

You could also run with debug logging - believe this logback.xml in the classpath will do it

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d %5p %40.40c:%4L - %m%n</pattern>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="console"/>
    </root>

    <logger name="com.couchbase" level="trace"/>"
</configuration>

@hgbalar, is it always bucket number 17 that fails?

@graham.pople mostly with 17th bucket also sometime happening with the 16th bucket. also checked that it’s not something wrong with that 16th or 17th bucket by verifying bucket name by reorder in list to connect it.

I observed it’s happening with memcached type bucket. it’s working for ephemeral type bucket

Ok thanks, just wanted to check if you were hitting some 16 limit somewhere (though I can’t think what that would be).

I’m not sure what’s going on here, sorry. You’ve already tried everything I would - increasing the timeout, checking if it’s always bucket 16, changing the order around to see if it’s a specific bucket. As Michael mentions, there should be no issues connecting to this many buckets, and his local testing confirms this.

Are you connecting to the buckets in serial or parallel?

Might be worth turning up the SDK logging and seeing if you can spot anything untoward in there.

connecting to the buckets in serial