{"id":4093,"date":"2017-10-30T09:05:19","date_gmt":"2017-10-30T16:05:19","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=4093"},"modified":"2025-06-13T15:55:15","modified_gmt":"2025-06-13T22:55:15","slug":"fast-failover-couchbase-server-5","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/fast-failover-couchbase-server-5\/","title":{"rendered":"Fast Failover with Couchbase Server 5.0"},"content":{"rendered":"
\n

Fast failover is one of the many improvements that come with the release of Couchbase Server 5.0<\/a> (now available for download).<\/p>\n<\/div>\n

\n

Failover is one of the important concepts to understand when it comes to distributed databases. The CAP theorem<\/a> states that a distributed database can\u2019t be both available and<\/em> consistent all<\/em> of the time. Couchbase Server\u2019s architecture is designed to be always<\/em> consistent, and partition tolerate. With fast failover, Couchbase Server is closing the gap on high availability<\/a>.<\/p>\n<\/div>\n

\n

In this blog post, I\u2019m going to demonstrate failover in action. I\u2019ll be using Docker to create a cluster of 3 Couchbase nodes on my local machine.<\/p>\n<\/div>\n

\n

You can follow along with the code sample in this blog post: it is available on GitHub<\/a>.<\/em><\/p>\n<\/div>\n

\n

Fast failover overview<\/h2>\n
\n
\n

You\u2019ll need a bit of setup and preparation.<\/p>\n<\/div>\n

\n

First, create a 3-node (at least) Couchbase Server cluster. There are a number of ways to do this, including Vagrants<\/a>, Virtual Machines, actual machines, Azure<\/a>, and more.<\/p>\n<\/div>\n

\n

I chose to use Docker. I blogged about how to create a Couchbase Cluster on Docker and access it with a .NET Core application<\/a> (don\u2019t forget the bridge network!). So, I just followed those same instructions again. The only difference is that I used a console application instead of an ASP.NET application (which you can read more about later in this post).<\/p>\n<\/div>\n

\n

\"Three<\/span><\/p>\n<\/div>\n

\n

I used the Couchbase Server 5.0.0-beta2 image from Docker Hub<\/a>, but by the time you read this, an official release of Couchbase Server 5.0 should be available on the official docker Couchbase repository<\/a>.<\/p>\n<\/div>\n

\n

Next, I created a bucket called “mybucket”. Make sure to enable replicas to create additional cop(ies) of data within the same cluster.<\/p>\n<\/div>\n

\n

\"Couchbase<\/span><\/p>\n<\/div>\n

\n

After that, create a user (I called mine “myuser”) with at least Data Writer and Data Reader permission for “mybucket”). If you aren\u2019t familiar yet with the Couchbase Server Role-based Access Control (RBAC)<\/a>, start with this blog post on Authentication with RBAC and .NET.<\/a><\/em><\/p>\n<\/div>\n

\n

\"Couchbase<\/span><\/p>\n<\/div>\n

\n

Finally, turn on automatic fast failover. From the Couchbase Console, go to Settings, and then Auto-Failover. Check the box to “Enable auto-failover”. As of version 5.0, you can set the Timeout value to as low as 5 (seconds). Previously, the value had to be at least 30 seconds.<\/p>\n<\/div>\n

\n

\"Enable<\/span><\/p>\n<\/div>\n

\n

There is a reason that auto-failover is off by default. Please review the full documentation on automatic failover<\/a> to make sure that it\u2019s a right fit for you.<\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n

.NET Example<\/h2>\n
\n
\n

Now that you have a 3-node cluster running inside of your Docker host, it\u2019s time to write a demonstration application. I decided to write a console application that would continuously perform reads against Couchbase. At some point, I will “pull the plug” on one of the nodes to show automatic fast failover in action.<\/p>\n<\/div>\n

\n

Connecting to the cluster<\/h3>\n
\n

After creating a new .NET Core console application in Visual Studio, I added the Couchbase .NET SDK (currently version 2.5.1) using NuGet<\/a>.<\/p>\n<\/div>\n

\n

Then, I created a configuration to connect to the 3-node cluster, authenticate to “myuser”, and open up “mybucket”.<\/p>\n<\/div>\n

\n
\n
var clientConfig = new ClientConfiguration\r\n{\r\n    Servers = new List<Uri>\r\n    {\r\n        new Uri(\"https:\/\/172.17.0.2\"),\r\n        new Uri(\"https:\/\/172.17.0.3\"),\r\n        new Uri(\"https:\/\/172.17.0.4\")\r\n    }\r\n};\r\nvar cluster = new Cluster(clientConfig);\r\n\r\nvar credentials = new PasswordAuthenticator(\"myuser\", \"password\");\r\ncluster.Authenticate(credentials);\r\n\r\n_bucket = cluster.OpenBucket(\"mybucket\");<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
Those IP addresses are the addresses that are internal<\/em> to the Docker host. This .NET Core application will also be running inside the Docker host, where those IP addresses will resolve. From outside<\/em> the docker host, only “localhost:8091” will resolve (assuming you are following the tutorial<\/a> I linked to earlier). If you are not using Docker, put in the IP addresses of the Azure machines, the VMs, etc, instead.<\/p>\n<\/div>\n
\n
Next, PasswordAuthentication<\/code> is used to ensure bucket access.<\/p>\n<\/div>\n
\n
Finally, get a bucket object using OpenBucket<\/code>.<\/p>\n<\/div>\n<\/div>\n
\n
Setting up documents<\/h3>\n
\n
For this demonstration, I want to setup a bunch of documents that I will later be reading from, repeatedly. First, I wrote a loop to create some arbitrary number of documents, that each have a key like “documentKey[num]” (e.g. “documentKey1”, “documentKey2”, etc).<\/p>\n<\/div>\n
\n
\n
var docKeys = new List<string>();\r\nfor (var i = 0; i < numDocuments; i++)\r\n{\r\n    var key = \"documentKey\" + i;\r\n    docKeys.Add(key);\r\n    _bucket.Upsert(key, new { name = \"Document\" + i });\r\n}<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
In my code, numDocuments<\/code> is set to 50. But if you are following along, feel free to set it to another number and see what happens.<\/p>\n<\/div>\n<\/div>\n
\n
Reading documents<\/h3>\n
\n
Therefore, there are 50 documents with well-known keys. The rest of the program will be continuously looping. Each loop iteration will attempt to retreive all 50 documents.<\/p>\n<\/div>\n
\n
\n
var iteration = 0;\r\nwhile (true)\r\n{\r\n    Console.WriteLine($\"Getting {numDocuments} documents [{iteration++}]\");\r\n    foreach(var docKey in docKeys)\r\n    {\r\n        var result = _bucket.Get<dynamic>(docKey);\r\n        if(terse)\r\n            ShowResultTerse(result, docKey);\r\n        else\r\n            ShowResult(result, docKey);\r\n    }\r\n    Console.WriteLine();\r\n\r\n    Thread.Sleep(2000);\r\n}<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
First, notice that there\u2019s a loop within the loop. The inner loop will run 50 times to perform a Get<\/code> on each document. ShowResult<\/code> will then output what\u2019s going on to the console (ShowResultTerse<\/code> does the same thing, just in a much more compact fashion. ShowResult<\/code> is below, but later screenshots will be using ShowResultTerse<\/code>).<\/p>\n<\/div>\n
\n
\n
private static void ShowResult(IOperationResult<dynamic> result, string id)\r\n{\r\n    \/\/ happy path, document was found\r\n    if (result.Success)\r\n    {\r\n        Console.WriteLine(\"Result: success\");\r\n        return;\r\n    }\r\n\r\n    \/\/ error, possibly node down\r\n    \/\/ show error, try to get replica\r\n    Console.WriteLine($\"Result: unsuccessful {result.Message}\");\r\n    Console.WriteLine(\"\\tAttempting to get replica.\");\r\n\r\n    var replica = _bucket.GetFromReplica<dynamic>(id);\r\n\r\n    \/\/ happy path for replica, it was found\r\n    if (replica.Success)\r\n    {\r\n        Console.WriteLine(\"\\tReplica result: success\");\r\n        return;\r\n    }\r\n\r\n    \/\/ error! replication may not be configured\r\n    \/\/ or it's possible something catastrophic happened\r\n    \/\/ this should be rare, but definitely want to log it\r\n    \/\/ maybe retry and\/or escalate\r\n    \/\/ in this example, it's just logged to console\r\n    Console.WriteLine(\"\\tReplica result unsuccessful: {result.Message}\");\r\n}<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
The comments will help you follow along, but ShowResult<\/code> does three checks:<\/p>\n<\/div>\n
\n
    \n
  1. Was the read successful? If so, output that. Done! Otherwise\u2026<\/li>\n
  2. Try to get a replica (from another node). Was THAT successful? If so, output that. Done! Otherwise\u2026<\/li>\n
  3. The application was unable to read the document or one of its replicas. In this example, that\u2019s going to be very rare. In reality, it could mean that the document doesn\u2019t exist, or replication isn\u2019t configured correctly, or something else has gone wrong.<\/li>\n<\/ol>\n<\/div>\n
    \n
    So, you\u2019re ready to run the application. If you\u2019re using Docker, don\u2019t forget to run this application in Docker (which is easy to do from Visual Studio<\/a>). (Also make sure to connect the .NET Core application container to the Docker bridge network).<\/em><\/p>\n<\/div>\n<\/div>\n
    \n
    Pull the plug!<\/h3>\n
    \n
    Before pulling the plug on one of the nodes, let\u2019s take a look at what the “normal” output is when running the above .NET Core application.<\/p>\n<\/div>\n
    \n
    In the below GIF, you\u2019ll see:<\/p>\n<\/div>\n
    \n