Connection failure for Couchbase behind Istio in EKS

Hi,

I have a Couchbase Cluster deployed in a EKS K8s cluster behind an Istio.

My main goal is to use an MSK (Kafka) Connect. It’s failing, (TimeoutException: Did not observe any item or terminal signal within 5000ms in 'source(MonoDefer)') so I was debugging using SDK Doctor.

I have in my K8s cluster the Couchbase Autonomous Operator and I defined the cluster using:

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: couchbase-cluster
spec:
  image: couchbase/server:7.1.3
  security:
    adminSecret: couchbase-cluster-auth
  networking:
    dns:
      domain: couchbase-cluster.couchbase-demo.io
    networkPlatform: Istio
  buckets:
    managed: false
  servers:
  - size: 3
    name: all_services
    services:
    - data
    - index
    - query
    - search
    - eventing
    - analytics

The Istio exposes the ports 9081-9085 and 11210 as follows:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-tool
  namespace: istio-routing
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
    - port:
        name: http
        number: 8091
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: xcdr
        number: 8092
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: query
        number: 8093
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: search
        number: 8094
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: analytics
        number: 8095
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: kv
        number: 11210
        protocol: TCP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: couchbase-virtual-service
spec:
  hosts:
    - couchbase-cluster.couchbase-demo.io
  gateways:
    - istio-routing/gateway-tool
  http:
    - match:
        - port: 8091
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8091
    - match:
        - port: 8092
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8092
    - match:
        - port: 8093
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8093
    - match:
        - port: 8094
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8094
    - match:
        - port: 8095
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8095
  tcp:
    - match:
        - port: 11210
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 11210

All those ports are open in the istio-ingressgateway as TCP.
After binding the ingress load balancer with the (private) DNS couchbase-cluster.couchbase-demo.io, I am able to query the cluster. E.g:

curl -v -X GET -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8091/pools/default/nodeServices 

curl -v -X POST -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8093/query/service --data 'statement=select meta().id from `beer-sample`.`_default`.`_default` data order by meta().id limit 2 offset 0' 

The output I get is:

[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)

When I run sdk-doctor from a pod inside the cluster, it works fine, so I think that the problem is that some networking setting is not right.

Does anyone know why it gets lost when it tries to access the couchbase-cluster-000*.couchbase-cluster.couchbase.svc ? What am I missing?

Thanks,

Manuel