Hi,
I have a Couchbase Cluster deployed in a EKS K8s cluster behind an Istio.
My main goal is to use an MSK (Kafka) Connect. It’s failing, (TimeoutException: Did not observe any item or terminal signal within 5000ms in 'source(MonoDefer)') so I was debugging using SDK Doctor.
I have in my K8s cluster the Couchbase Autonomous Operator and I defined the cluster using:
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: couchbase-cluster
spec:
image: couchbase/server:7.1.3
security:
adminSecret: couchbase-cluster-auth
networking:
dns:
domain: couchbase-cluster.couchbase-demo.io
networkPlatform: Istio
buckets:
managed: false
servers:
- size: 3
name: all_services
services:
- data
- index
- query
- search
- eventing
- analytics
The Istio exposes the ports 9081-9085 and 11210 as follows:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: gateway-tool
namespace: istio-routing
spec:
selector:
app: istio-ingressgateway
istio: ingressgateway
servers:
- port:
name: http
number: 8091
protocol: HTTP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
- port:
name: xcdr
number: 8092
protocol: HTTP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
- port:
name: query
number: 8093
protocol: HTTP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
- port:
name: search
number: 8094
protocol: HTTP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
- port:
name: analytics
number: 8095
protocol: HTTP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
- port:
name: kv
number: 11210
protocol: TCP
hosts:
- '*.couchbase-demo.io'
- couchbase-demo.io
- ./*
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: couchbase-virtual-service
spec:
hosts:
- couchbase-cluster.couchbase-demo.io
gateways:
- istio-routing/gateway-tool
http:
- match:
- port: 8091
route:
- destination:
host: couchbase-cluster
port:
number: 8091
- match:
- port: 8092
route:
- destination:
host: couchbase-cluster
port:
number: 8092
- match:
- port: 8093
route:
- destination:
host: couchbase-cluster
port:
number: 8093
- match:
- port: 8094
route:
- destination:
host: couchbase-cluster
port:
number: 8094
- match:
- port: 8095
route:
- destination:
host: couchbase-cluster
port:
number: 8095
tcp:
- match:
- port: 11210
route:
- destination:
host: couchbase-cluster
port:
number: 11210
All those ports are open in the istio-ingressgateway as TCP.
After binding the ingress load balancer with the (private) DNS couchbase-cluster.couchbase-demo.io, I am able to query the cluster. E.g:
curl -v -X GET -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8091/pools/default/nodeServices
curl -v -X POST -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8093/query/service --data 'statement=select meta().id from `beer-sample`.`_default`.`_default` data order by meta().id limit 2 offset 0'
The output I get is:
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
When I run sdk-doctor from a pod inside the cluster, it works fine, so I think that the problem is that some networking setting is not right.
Does anyone know why it gets lost when it tries to access the couchbase-cluster-000*.couchbase-cluster.couchbase.svc ? What am I missing?
Thanks,
Manuel