we have a cluster made of 5 nodes. 3 of them are used for data and 2 of them are used for index and query services.
Each group of nodes are deployed into an AWS Autoscaling Group. We have 2 autoscaling groups.
The problem that AWS Autoscaling groups presents, is that nodes from Autoscaling group have not any fixed IPs.
That issue make not possible to assign automatically any DNS entry to the nodes IPs (just because, the IPs from the nodes are ephemeral). Because of that, we thought about setting a load balancer between the application and the cluster.
AFAIK we cannot put a load balancer between the application and the cluster, but it seems that there is not any possibility to set dinamically the DNS entries to the new IPs for the nodes that enter to make part of the cluster.
So my questions are:
In this specific case, Could we set a load balancer between our Autoscaling group and our application?
In case we cannot have a load balancer, Is there any solution to dinamically set DNS entries to the new IPs from the nodes that the cluster is made of?
Which is the best solution to integrate a couchbase cluster into an AWS Autoscaling group?
I’m going to tag @btburnett3 on this one. I know they are using Couchbase and AWS, and he also worked on a .NET extension for DNS SRV, so he might have a suggestion or two (no promises though, he is a busy man
There are two different ways I know to address this problem, both with pros and cons.
Create a load balancer that balances port 8091 only and which is linked to your auto scaling group. Then point the SDK to the load balancer, and configure the SDK to use HTTP bootstrapping only (no CCCP streaming). The SDK will get the node information over port 8091 from the load balancer, then all subsequent communication will be directly to the nodes. The downside is that the HTTP bootstrap method is slower, and at large cluster scales can also create problems. This isn’t the Couchbase recommended approach, but it does work. Here is a blog I wrote, the beginning of which covers this approach https://blog.couchbase.com/using-couchbase-aws-kubernetes-microservices/.
The better option is using DNS SRV records. As you mentioned, however, automatic registration is the problem. I have worked around this by using Lambda functions to perform the registration automatically when the auto scaling group changes. I created a reusable Terraform module that does this, or you could create the Lambda manually.