To ensure data safety you need to ensure there are enough nodes within the cluster to support the safety requirements for your data. This involves retaining a suitable number of nodes, and node configuration, within your cluster. There are two aspects to consider, the distribution of information across your nodes, and the number of replicas of information stored across your cluster.
The basic idea is that more nodes are better than less. If you only have 2 nodes, your data will be split across the two nodes half, and half. This means that half of your dataset will be "impacted" if one goes away. On the other hand, with 10 nodes, only 10% of the dataset will be "impacted" if one goes away. Even with Automatic Failover, there will still be some period of time when data is unavailable if nodes fail. This can be mitigated by having more nodes.
You also need to take into account the amount of extra load that the cluster will need to take on after a failover. Again, with only 2 nodes, each one needs to be ready to handle the entire load. With 10, each node only needs to be able to take on an extra 10th of the workload should one fail.
While 2 nodes does provide a minimal level of redundancy, it is recommend always using at least 3 nodes.
Couchbase Server allows you to configure up to 3 replicas (creating 4 copies of the dataset). In the event of a failure, you can only "failover" (either manually or automatically) as many nodes as you have replicas. For example:
In a 5 node cluster with one replica, if one node goes down, you can fail it over. If a second node goes down, you no longer have enough replica copies to fail over to and will have to go through a slower process to recover.
In a 5 node cluster with 2 replicas, if one node goes down, you can fail it over. If a second node goes down, you can fail it over as well. Should a 3rd one go down, you now no longer have replicas to fail over.
After a node goes down and is failed over, you should attempt to replace that node as soon as possible and rebalance. The rebalance is what will recreate the replica copies (if you still have enough nodes to do so).
As a rule of thumb, it is recommended that you configure:
1 replica for up to 5 nodes
1 or 2 replicas for 5 to 10 nodes
1, 2 or 3 replicas for over 10 nodes
While there many be variations to this, there are definitely diminishing returns from having more replicas in smaller clusters.
In general, Couchbase Server has very low hardware requirements and is designed to be run on commodity or virtualized systems. However, as a rough guide to the primary concerns for your servers:
RAM: Your primary consideration as RAM is used to keep active items and is the key reason Couchbase Server has such low latency.
CPU: Couchbase Server has very low CPU requirements. The server is multi-threaded and therefore benefits from a multi-core system. Machines with at least 4 or 8 physical cores are recommended.
Disk: By decoupling the RAM from the IO layer, Couchbase Server can support low-performance disks better than other databases. Known working configurations include SAN, SAS, SATA, SSD, and EBS, with the following recommendations:
SSDs have been shown to provide a great performance boost both in terms of draining the write queue and also in restoring data from disk (either on cold-boot or for purposes of rebalancing).
RAID generally provides better throughput and reliability.
Striping across EBS volumes (in Amazon EC2) has been shown to increase throughput.
Network: Most configurations will work with Gigabit Ethernet interfaces. Faster solutions such as 10GBit and Inifiniband will provide spare capacity.
Due to the unreliability and general lack of consistent IO performance in cloud environments, we highly recommend lowering the per-node RAM footprint and increasing the number of nodes. This will give better disk throughput as well as improve rebalancing since each node will have to store (and therefore transmit) less data. By distributing the data further, it will make the impact of losing a single node (which could be fairly common) even less.
Also, see the best practices for Section 4.6, “Using Couchbase in the Cloud”.