Auto scaling, DNS, and EBS volumes
This is something I've been trying to figure out for a little while now. I'm deploying an array of Membase servers inside of Rightscale. I've got the array configured to have all configuration settings and data on an EBS volume that I take regular snapshots of. So far everything is pretty cut and dry and works well. Where I start to have problems is when I mix auto scaling with DNS. I want the array to be able to auto scale, but that causes problems with the MB_DNS_NAME input. Right now I'm not aware of how to create a new A record in DNSMadeEasy automatically, nor am I aware of how to get the DDNS ID on the fly if the first part was possible so when a new server spins up from the auto scaling feature, I don't have a value for the MB_DNS_NAME. The work around that I'm using right now is I'm using the EC2_HOSTNAME of the server as the MB_DNS_NAME. Doing this gives me the ability to auto scale the array and dynamically assign a DNS name that resolves to that particular server. The only problem with this is that it's causing problems when I try to create a new server from the latest snapshot. Since any new server that I launch has a new EC2_HOSTNAME, its creating a new EBS volume instead of restoring from an existing snapshot. I tried using the EBS_LINEAGE_PREFIX_OVERRIDE input while setting the INITIAL_LAUNCH value to false, but when the server was operational I could talk to it. All I would get is errors that the IP has changed when I look at the new server in the web console.
So would anyone have any suggestions on how to configure an auto-scalable array of Membase servers, with or without using DNSMadeEasy, that's still able to utilize EBS volumes for backup purposes?
Hi Mike, thanks for reaching out.
At the moment, it's not possible to combine the use of a DNS name with auto-scaling within RightScale. It's an issue we're aware of, but it's not yet clear how to effectively resolve it (as you've noticed, you can't pass different inputs to each server of an array and so it's very hard for each server to do anything differently from the others).
My recommendation would be to leave the MB_DNS_NAME blank (the default) within your autoscaling group. The main purpose of supplying a DNS name is so that if an instance goes down, the same server could be relaunched and automatically assume the identity of the one that was lost. By using a DNS name as a logical identifier between nodes of a cluster, the underlying IP address is free to change (as it will when an instance gets relaunched). This greatly aids in the recovery process should an instance terminate unexpectedly.
When not using a DNS name, the recovery process is a bit more manual, but not impossible. This is where we're looking to make it better in the future, but you'd have to manually identify which EBS volumes were assigned to the cluster, remount them and restore the data into a newly created cluster. While admittedly not the smoothest process, you can rest assured that the data has not been lost.
If you're intimately familiar with the internals of RightScale, I'm all ears on how to better approach this. We can do script lots of things that the user could do manually, but it's currently quite difficult to identify a "set" of EBS volumes that were in use at a specific point in time from which to recover from.
On a slight tangent, we have had success with customers combining a static deployment with an autoscaling group. You would setup a "base cluster" of the minimum number of nodes that you will need, and then create an autoscaling group on top of that. Although I can see a little bit of confusion when doing so, I would recommend using DNS names for that base cluster but not for the autoscaling group.
Let me know if you have any other questions or concerns.
Perry
Sorry you've not had much luck with an answer thus far Mike... I just sent this to a couple of people who can hopefully jump in and help.