Hello Good Morning. I am trying to establish connectivity from an client Node to Couchbase server. Couchbase server and the kafka connect client node run on different azure virtual networks however these virtual networks are peered. Couchbase version is 6.6 and Kafka Connect versions is 4.1 and runs in OS Ubuntu. Now the issue is while trying to use DNS names instead of individual IP’s in the Kafka Couchbase Source Connector Setting(couchbase.seed.nodes). I will post the source connector details below. The kafka source connector successfully establishes connectivity with individual IP however is not able to resolve DNS and give below error
Error Details
[2021-10-15 14:56:14,779] INFO [com.couchbase.core][DnsSrvLookupFailedEvent][172ms] DNS SRV lookup failed (name not found), trying to bootstrap from given hostname directly. (com.couchbase.core:537)
[2021-10-15 14:56:14,779] INFO [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.2.0”,“clientGitHash”:"${buildNumber}",“coreVersion”:“2.2.0”,“coreGitHash”:"${buildNumber}",“userAgent”:“couchbase-java/3.2.0 (Linux 5.8.0-1042-azure amd64; OpenJDK 64-Bit Server VM 11.0.11+9-Ubuntu-0ubuntu2.20.04)”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:4,“eventLoopGroups”:[“EpollEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“eventingCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:4500,“configIdleRedialTimeoutMs”:300000,“memcachedHashingStrategy”:“StandardMemcachedHashingStrategy”},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:false,“nativeTlsEnabled”:true,“hostnameVerificationEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:null,“ciphers”:},“timeoutConfig”:{“kvMs”:2500,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:75000,“viewMs”:75000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:30000,“disconnectMs”:10000,“eventingMs”:75000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:true,“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“enabled”:true},“thresholdLoggingTracerConfig”:{“enabled”:true,“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024,“kvThresholdMs”:500,“queryThresholdMs”:1000,“searchThresholdMs”:1000,“analyticsThresholdMs”:1000,“viewThresholdMs”:1000},“loggingMeterConfig”:{“enabled”:true,“emitIntervalMs”:600000},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“ThresholdLoggingTracer”,“meter”:“LoggingMeter”,“numRequestCallbacks”:0} {“coreId”:“0xbd7437d00000007”,“seedNodes”:[{“address”:“couch-xxx.xx.xxxx.com”}]} (com.couchbase.core:537)
[2021-10-15 14:56:14,779] INFO [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0xbd7437d00000007”,“managerPort”:“8091”,“remote”:" couch-xxx.xx.xxxx.com"} (com.couchbase.node:537)
[2021-10-15 14:56:44,768] WARN [com.couchbase.endpoint][EndpointConnectionFailedEvent][30s] Connect attempt 1 failed because of TimeoutException: Did not observe any item or terminal signal within 30000ms in ‘source(MonoDefer)’ (and no fallback has been configured) {“circuitBreaker”:“DISABLED”,“coreId”:“0xbd7437d00000007”,“remote”:" couch-xxx.xx.xxxx.com:11210",“type”:“KV”} (com.couchbase.endpoint:567)
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 30000ms in ‘source(MonoDefer)’ (and no fallback has been configured)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:294)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:279)
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:418)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:270)
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:285)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
[2021-10-15 14:56:44,768] WARN [com.couchbase.endpoint][EndpointConnectionFailedEvent][30s] Connect attempt 1 failed because of TimeoutException: Did not observe any item or terminal signal within 30000ms in ‘source(MonoDefer)’ (and no fallback has been configured) {“bucket”:“xxx”,"circuit
Kafka Source Connector Details
sudo curl -X POST http://xx.xx.xx.xx:7088/connectors -H “Content-Type: application/json” -d @- <<BODY
{
“name”: “xxxxxxxx”,
“config”: {
“name”: “xxxxxxx”,
“connector.class”: “com.couchbase.connect.kafka.CouchbaseSourceConnector”,
“tasks.max”: 2,
“couchbase.topic”:“xxxxxxxxx”,
“couchbase.seed.nodes”:“xxxxxxx.com” ,
“couchbase.bootstrap.timeout”:“30s”,
“couchbase.bucket”:“xxxxxx”,
“couchbase.username”:“xxxxxxx”,
“couchbase.password”:“xxxxxxx”,
“use_snapshots”: “false”,
“couchbase.event.filter”:“com.couchbase.connect.kafka.filter.AllPassFilter”,
“couchbase.source.handler”: “com.couchbase.connect.kafka.handler.source.DefaultSchemaSourceHandler”,
“couchbase.stream.from”:“SAVED_OFFSET_OR_NOW”,
“couchbase.compression”:“ENABLED”,
“couchbase.flow.control.buffer”:“128m”,
“couchbase.persistence.polling.interval”:“100ms”,
“couchbase.xattrs”:true
}
}
BODY
NSLOOKUP SRV records
dig srv xxxx.com
; <<>> DiG 9.16.1-Ubuntu <<>> srv xxxx.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53818
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;xxxx.com. IN SRV
;; ANSWER SECTION:
xxxx.com. xx IN CNAME xxxx.com.
;; Query time: 144 msec
;; SERVER: xx.0.0.53#53(xx.0.0.53)
;; WHEN: Fri Oct 15 15:16:48 UTC 2021
;; MSG SIZE rcvd: 79