I’ll address #2 first as it’s easiest. We are releasing 1.0.0 in very soon which will have support for XDCR across a site-to-site VPN. The essential things that need to work are that you have connectivity from one Kubernetes cluster to the other e.g. they need to have non-overlapping address ranges, the necessary routes installed via the VPN and finally one Kubernetes node in one region can ping another node in another region. There will be a new Couchbase Operator cluster option introduced called exposedFeatures; add “xdcr” to this and you will be able to establish the XDCR connection via a node port IP address. AKS should have VNet support to create the VPN.
Addressing #1, there is no easy way to perform this integration at present, although our clients are being updated to take advantage of the mechanism which allows XDCR to function (this should begin to be released in a few months time), again this would require a VPN from the App Service to the Kubernetes network, and the use of a node port IP address.
Usually we’d expect applications to be deployed in Kubernetes along side the Couchbase cluster and make use of the built in service discovery provided by Kubernetes’ network layer, hence my interest in your use case.
I also need to connect external SDKs to a Couchbase cluster in Kubernetes.
In my situation, AWS Lambdas should be able to connect to the Couchbase cluster in Kubernetes.
I’m sure that because of this use case external SDKs should be supported.
Thanks for your feedback. There a couple deficiencies with Kubernetes which makes this use pattern less than perfect that you should be aware of.
We could need to expose ports externally to the internet with a Service per Couchbase node. As there is no provision for control over external DNS we’d have to expose the ports via IP only. During the life-cycle of the cluster, those IP addresses would change and eventually your lambda functions would cease to function.
Additionally you need to consider TLS as this is running over the public internet. Having no DNS makes this somewhat tricky as you’d have to supply us with a CA certificate and key so we could create server certificates with the relevant IP alternative names.
I guess all the problems go away if we were to be granted control over a DNS server so we could control A and SRV records via DDNS. Actually a quick google suggests you are not the first with this requirement: https://github.com/kubernetes-incubator/external-dns.
So I believe this pattern will be possible at some point in the future but not immediately. We’re still left with the problem that the external-dns controller probably won’t support aggregating individual services into a single SRV record, so your Couchbase connection string will be prone to failure as names change. I’ll have a word with the developers and see if it’s realistic.
Does this issue affect use of the Couchbase Kafka connector?
We are running CB in K8s using the Autonomous Operator. While testing this using Docker for Windows, I have had no success connecting an application from the host or from a docker container to the CB cluster in K8s. Now I understand why. (would be worth documenting this limitation on the Autonomous Operator Docs, its cost me hours).
While this is not an issue for our own part of the application as that will run within the K8s cluster, we also need to run a kafka service. Until confluent release their autonomous operator I was planning on running kafka in a docker container and connecting that to the CB cluster in K8s.
Will the kafka connector be able to connect to the CB cluster in K8s? Does the kafka connector use the SDK?
So 8 months is a long time in the world of Kubernetes, and we’ve made a lot of progress that I will share here as its relevant to the thread in general. 1.2.0 is out in around a month. What we have done is reuse the multi-network configuration stuff in Couchbase Server that is used to enable XDCR via IP and add in support for DNS names. Coupled with public addressability as a result of optionally using LoadBalancer rather than NodePort services we are now able to provide connectivity over the public internet, with full end-to-end TLS encryption.
As regards SDKs they should be able to connect using this mechanism from anywhere in the universe. I’m no expert on the Kafka connector but I can see it’s written in Java so highly likely uses the Java SDK under the hood.
I’ll revisit the networking documentation before 1.2.0 and ensure it’s very clear on networking options, never the easiest topic to describe
In the mean time if it’s holding up your testing, you can add client to the spec.exposedFeatures attribute. It’s up to you to ensure there is pure L3 routing (absolutely no DNAT, the IP address client connects to must be the same as what Couchbase Server advertises) between your Kafka container and the Kubernetes underlay network, a site-to-site VPN is usually the easiest to configure. Then to connect the Kafka connector to Couchbase Server follow the guide here https://docs.couchbase.com/operator/1.1/xdcr.html#using-ip-and-exposed-features.
You cannot use an Ingress for Couchbase as that only supports ports 80 and 443. Our ports are 8091, 8092… We also can expose up to a maximum of 6 different ports, these cannot be multiplexed over a single port 443 (the data service is a binary protocol for starters!), so a Service is the only option.
You can use anything you want to provide TLS certificates, the only constraints we need are the correct wildcard X.509 subject alternate names so clients can match the host name they are connecting to with the hosts the certificate is valid for.
You connect in exactly the same way as XDCR e.g. client connect to connect to http://x.y.z.w:3xxxx where x.y.z.w is a node IP and 3xxxx is the node port.
Now the problem with using node ports is that if a node goes away or changes address, the clients will break. If the pod that generates the node port goes away, the clients will break. As you are using IP addresses and NodePorts you cannot encrypt the traffic. Be aware of these limitations.
The correct way to connect will be described in the upcoming Operator 2.0 documentation. The short version is that your clients talk to a DNS server that forwards the DNS zone %namespace%.svc.cluster.local to the remote Kubernetes DNS server where the cluster lives. The remote cluster must be using flat networking (no overlays). The client can then connect to couchbase://%clustername%.%namespace%.svc (and must have at least a cluster.local search domain configured for its stub resolver). This gives you high-availability, service discovery and the option of using TLS.
Thanks for your reply Simon. Will that approach work where access is required to a range of Couchbase ports (i.e. via the Java client library)? The example you gave of XSDR only requires one port, I beleive.