Best practice around using Membase on Amazon EC2
Hi,
We're looking to deploy a 4-6 node Membase cluster on Amazon EC2 and would like to know what are the best practices and guidelines beyond what's mentioned in the Using Membase in the Cloud article?
Use of EBS Volumes
We're using Windows Server 2008 for the nodes which from what I understand is backed and run directly from an attached EBS volume, so am I correct to assume that we won't need to create dedicated EBS volumes in order to persist the data beyond the lifetime of the instances (so long that we don't terminate the instances manually or allow the auto-scaler to scale down the cluster, both of which will delete the attached EBS volume)?
Determining the number of replicas
Is there a recommended setting for the number of replicas?
If I set the number of replicas to 3 for each of say, 10 membase buckets in a 6 node cluster, how will those replicas be distributed amongst the 6 available nodes seeing as there's no way to select the replica nodes from the web console?
Will the same nodes hold the replica for each of the 10 replicated buckets or will each bucket have a different set of replica nodes?
If one of the nodes holding the replica for one of the buckets is removed from the cluster, will another node take over the responsibility of holding the replica and therefore maintaining the number of replicas in the cluster? Or will the number of replica for that bucket be reduced by 1?
Conversely, what happens when the removed node is added back into the cluster?
Is it possible to increase the number of replica for an existing bucket?
Removing a node
Assuming that all our buckets are membase buckets, persisted and replicated across multiple nodes, based on what I read on another thread, in order to remove a node from the cluster it's best to 'remove' the server as opposed to just 'fail over' because it allows any pending replication to complete.
Is there anything else we should be aware of when scaling down a cluster?
Backing up and Restore
I understand that right now there's no easy way to create a backup of the cluster beyond copying all the .sqlite files from all the nodes in the cluster, but if all the buckets are replicated, does it not mean that I just need to take one copy of the data for each replicated bucket?
Sorry this ended up being such a long question and many thanks in advance for any insight you might be able to offer!
Cheers,
Hi there, thanks for your quetions...even if it was a long one!
1 - EBS Volumes: Yes, if you're using EBS-boot then there is no need to have a separate EBS volume. I would highly recommend testing this very thoroghly to make sure the data can be retrieved if the instance goes away for whatever reason (or if the whole cluster dies).
2 - There's no real recommendation on setting the number of replicas. In general, you can sustain the loss of as many servers at once as you have replicas (1 replica means you can lose 1 server at a time, 2 replicas mean you can lose up to 2 servers at a time, etc).
Within a Membase cluster, all buckets and all replicas are spread evenly throughout the cluster. For each bucket, we divide the keyspace up into 1024 vbuckets (http://wiki.membase.org/display/membase/vBuckets) and each of those vbuckets is replicated as many times as you have configured. The replicas are spread throughout the cluster so that no replica lives on the same node as it's "active" copy. Obviously you'll need at least as many servers as you have configured replicas.
In your example, each node will hold 1 or more (many more) vbuckets for all buckets as well as 1 or more (many more) replica vbuckets from other nodes.
If a node is removed AND the cluster is rebalanced, then new replicas will be created among the existing nodes (provided there are enough). If a node is simply failed over, no replicas are recreated until the cluster is rebalanced. When adding nodes, both the active vbuckets and replica vbuckets are redistributed to make the newly sized cluster balanced.
It is not currently possible to change the number of replicas for an existing bucket.
3 - Removing a node. Yes, you are correct that you should use the "remove and rebalance" workflow rather than failover to properly remove a node. That should be all you need to be concerned with when scaling down a cluster.
4 - Backup up and Restore. You'll really need to take a backup of all the sqlite files on all nodes since both the active and replica data is spread across all nodes. Keep in mind that any cluster-wide backup is only valid for that particular cluster topology. If you've made changes to the toplogy (removing/adding nodes) any previous backup is no longer valid (unless you're restoring it back in time). Also, make sure you backup the configuration files as well. We are working on a formal backup/restore tool.
Hope that answered your questions, feel free to follow up for more clarification.
Thanks,
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!