{"id":1879,"date":"2021-03-23T01:24:55","date_gmt":"2021-03-23T08:24:55","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/distributed-multi-document-acid-transactions\/"},"modified":"2021-03-23T01:24:55","modified_gmt":"2021-03-23T08:24:55","slug":"distributed-multi-document-acid-transactions","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/distributed-multi-document-acid-transactions\/","title":{"rendered":"How we implemented Distributed Multi-document ACID Transactions in Couchbase"},"content":{"rendered":"\n

ACID Transactions<\/a> are a must when you have strict data consistency requirements in your application. The costs of running transactions on distributed systems can rapidly create bottlenecks at scale. In this article, we will give you an overview of some of the challenges faced by NoSQL and NewSQL databases. Then, we’ll deep dive into how Couchbase implemented a scalable distributed transaction model with no central coordination and no single point of failure. Additionally, I will also give a short overview of how the support for transactions in N1QL looks like on Couchbase 7.0.<\/span><\/p>\n\n\n\n

Some minor details were omitted for the sake of simplicity.<\/span><\/p>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Relational vs NewSQL vs NoSQL Transactions<\/span><\/h2>\n\n\n\n

Before I start explaining how Couchbase implemented support for transactions, I need to first explain the inherent characteristics of atomicity in relational and NoSQL databases (using semi-structured data models like JSON):<\/span><\/p>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Atomicity in RDBMS<\/span><\/h3>\n\n\n\n

Let’s say you need to save a new user in the database. Naturally, as it has many other tables associated with it, inserting a user will also require inserts on many other tables:<\/span><\/p>\n\n\n\n

\"Transactions<\/p>\n\n\n\n

Because the relational model forces you to store everything on \u201cboxes\u201d and split your data into small pieces, adding a new user should always run inside of a transactional context. Otherwise, if one of your inserts fail, your user will end up half-saved. Notice how an RDBMS relies heavily on transactions, as applications are far more complex than when the relational model was originally designed back in the seventies.<\/span><\/p>\n\n\n\n

Luckily, as these databases are designed to run on a single node, you can use a central transaction coordinator to commit the data at once without any performance impact. <\/span><\/p>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Atomicity in NewSQL<\/span><\/h3>\n\n\n\n

On the NewSQL side (distributed relational), things are a little bit more complicated. As most of these databases reuse the relational model, your entity\u2019s data (or <\/span>aggregate root<\/span><\/a>) tends to be spread throughout multiple nodes.<\/span><\/p>\n\n\n\n

\"atomicity<\/p>\n\n\n\n

In the image above, if we need to load the user into memory, we would need to first get the user from Server 1, then load the association between users and roles on Server 2 and finally load the target role from Server 3. This simple operation requires data to travel at least twice over the network, which will ultimately limit your read performance. In a real-world scenario, a user has many more tables associated with it. This is why distributed relational is not yet practical when you need to read\/write as fast as possible.<\/span><\/p>\n\n\n\n

You can try to minimize the issues above by limiting the size of your cluster, by relying heavily on indexes to track all relationships, or by some sharding techniques to keep all related data in the same node (which is difficult to implement in practice). The last two approaches, even when well implemented, will consume significant resources from the database to be managed properly.<\/span><\/p>\n\n\n\n

ACID transactions in NewSQL databases require more coordination than in NoSQL, as the data related to an entity is split into multiple tables which might live in different nodes. The relational model, as we use today, requires transactions for the majority of the writes, updates, and cascading deletes. The extra coordination that the NewSQL architecture requires comes at a cost of reduced throughput for applications requiring low latency operations.<\/span><\/p>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Atomicity in Document Databases<\/span><\/h3>\n\n\n\n

The use of semi-structured data like JSON can drastically reduce the number of \u201ccross-node joins\u201d, therefore delivering better read\/write performance without the need to rely too much on indexing. This was one of the key insights of the <\/span>Dynamo Paper<\/span><\/a> (first published ~13 years ago) which was the<\/span> catalyst to create the NoSQL databases as we know them today.<\/span><\/p>\n\n\n\n

Another interesting characteristic of a semi-structured data model is that it is less transactional, as you could fit the whole user data in a single document:<\/span><\/p>\n\n\n\n

\"atomicity<\/p>\n\n\n\n

As you can see in the image above, the user preferences and roles can easily fit inside a \u201cUser Document\u201d, so there is no need for a transaction to insert or update a user as the operation is atomic. We insert the document or the whole operation fails. The same is valid for many other common use cases: shopping carts, products, tree structures, and <\/span>aggregate roots <\/span><\/a>in general.<\/span><\/p>\n\n\n\n

In the majority of the applications using document databases, 90% of the transactional operations will fall in this single document category. But\u2026 what about the other 10%? Well, for those we will need multi-document transaction support, which has been added in Couchbase since version 6.5 and is the main focus of this article.<\/span><\/p>\n\n\n\n

Here is a presentation about transactions that was delivered at Couchbase Connect 2020. Matt Ingenthron gives you an explanation of when and why you might need multi-document ACID transactions:<\/span><\/p>\n\n\n\n

Watch the full version at <\/span>https:\/\/www.youtube.com\/watch?v=2fsZVe2cT3M&ab_channel=Couchbase<\/span><\/a><\/h6>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Video Transcript<\/span><\/h3>\n\n\n\n
\nClick to read the full transcription.<\/summary>\n

Let’s talk about how this is applied to a fictitious but maybe kind of realistic example of a document model for a Couchbase system. So we’ve raised some money, and we’re going to build a massively multiplayer online role-playing game (an MMORPG), with players and monsters. So we need a data model.\u00a0<\/span><\/p>\n

We are going to have players who fight monsters and then based on that fight, win or lose, they’re going to. If they win, they’re going to get a weapon, if they lose, uh, they lose some hit points so nobody can die. You always can come back to life, you can always find another day. But your players fight monsters, and we get our version 1.0 built. Great! Okay, Got our funding, got our 1.0 built.<\/span><\/p>\n

The problem is, we forgot to do the massively multiplayer part. There’s no collaborative play. I can’t have multiple players fight the same monster. So I need to fix that, right? So let’s roll out a new version. So players are going to continue to fight monsters and win weapons.<\/span><\/p>\n

So we roll out our version two 2.0, and in version 2.0 players can fight monsters together. I can coordinate with my friends, we could go find a monster, and we can kill that monster.\u00a0<\/span><\/p>\n

But we left a bug in there. It’s possible for multiple players to deliver the death blow and the reason that’s a problem is the players of the game figure it out. Instead of battling a monster together to death, what they do is, they, in this massively multiplayer world, they\u2019ll battle it till close to death, and then a bunch of players will gather up together, and they all deliver the death blow or many will deliver the death blow at the same time.\u00a0<\/span><\/p>\n

The problem is, they win items, they win multiple items, and because items have rarity, and if you don’t have a certain amount of rareness for an item, the gameplay is not very interesting. This bug has allowed too many items to exist in the world, and the players are just spending time hacking the game, and then they get bored and leave. We need to keep the gameplay interesting.<\/span><\/p>\n

So let’s think about this. How can we fix this? I think what we’ll need to do is probably introduce a fix. We\u2019ll still allow players and monsters, multiple players to fight a monster. But what we’re gonna do is take one of those tricks that we have, we’re gonna take <\/span>Couchbase CAS <\/span><\/a>Operations.<\/span><\/p>\n

With the <\/span>CAS operation<\/span><\/a> what happens is now multiple players are fighting that monster, lowering its hit points until it gets down to zero. But only one of those players is going to be able to deliver the death blow to that monster and win an item.<\/span><\/p>\n

So the way this works is if two players are trying to deliver that death blow, the application <\/span>server that’s handling the request is going to try to modify the document. It has to pick up a little piece of opaque information that we call the<\/span> CAS (that’s for Check And Set)<\/span><\/a>, and so that means that if that opaque does not match, then the documents have already been modified, and so you need to retry that operation. In the scenario that two actors within the system are trying to grab that document at the same time, what we want is one to succeed and one to fail, and the CAS will give us that in a very efficient way.<\/span><\/p>\n

We pull out that trick, we introduce CAS operations, the bug is fixed, gameplay is now much more interesting and 2.1 does really well, so that’s great.<\/span><\/p>\n

So now let’s try a , uh, we wanna move on, we wanna make things more interesting. Imagine now that I introduce another feature: “Players can still fight monsters together, but they have to do it outside the city. So you have to be outside the city wall where the players are, and if you go into the city, you will do trade in a bizarre”<\/span><\/p>\n

This works really well at first, but then players figure something out.<\/span><\/p>\n

So imagine player1, I have to retrieve the document for player1. Then I have to retrieve the document for player2. Then I have to move the sword from player1 to player2, that’s very easy to do in the application logic, and then I go to store that change back to the system with the CAS operation, and then I go to store the other change back, right? Sounds like it’ll be great. Except there’s a bug.<\/span><\/p>\n

The bug here is that my players can start a trade and then disconnect, and then items that might we that we might want to be rare are not going to be rare. They can be duplicated within the system.<\/span><\/p>\n

In massively multiplayer online role-playing games, it’s called a Dup Bug. If you were to go to Google and search search for a “Dup Bug”, you’ll find lots of scenarios.<\/span><\/p>\n

Here’s one from just a couple of weeks ago where Final Fantasy Crystal Chronicles on switch had to be patched because of a Dup bug. And then there was one just a few days before that, this is from a blog where a game blogger showing people how to use this dup glitch to retrieve… to get additional items within the game.<\/span><\/p>\n

So we need to fix this bug. How are we going to do that? Well, so we’re going to reach into our Couchbase bag of tricks.<\/span><\/p>\n

We’re going to introduce Couchbase transactions. The gameplay is almost exactly the same. But what are we gonna do with Couchbase transactions? And so this is the one slide where I’m going to talk a little bit about the code.<\/span><\/p>\n<\/details>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Multi-Document Distributed ACID Transactions in Couchbase<\/span><\/h2>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Now that you understand how transactions behave in different data models, it is time to deep dive into how we have implemented it at Couchbase and what led to our design choices. First, let\u2019s go through the syntax:<\/span><\/p>\n\n\n

[crayon lang=”java” decode=”true”]transactions.run((ctx) -> {
\n \/\/ get the account documents for userA and UserB
\n TransactionJsonDocument userA = ctx.getOrError(collection, “userA”);
\n JsonObject userAContent = userA.contentAsObject();
\n int userABalance = userAContent.getInt(“account_balance”);
\n TransactionJsonDocument userB = ctx.getOrError(collection, “Beth”);
\n JsonObject userBContent = userB.contentAsObject();
\n int userBBalance = userBContent.getInt(“account_balance”);<\/p>\n

\/\/ if userB has sufficient funds, make the transfer
\n if (userBBalance > transferAmount) {
\n userAContent.put(“account_balance”, userABalance + transferAmount);
\n ctx.replace(userA, userAContent);
\n userBContent.put(“account_balance”, userBBalance – transferAmount);
\n ctx.replace(userB, userBContent);
\n }
\n else throw new InsufficientFunds();
\n});[\/crayon]<\/p>\n\n\n\n

The Java example above is the classic example of how to transfer money between two clients.\u00a0 Notice that we decided to use a <\/span>lambda function<\/span><\/a> to express the transaction. Proper error handling can be challenging in this scenario and wrapping your transaction with an anonymous function allows the Couchbase Java SDK to do that work for you (i.e retry if something fails).<\/span><\/p>\n\n\n\n

When we first launched support for transactions we were trying to avoid verbosity. This was how one competitor’s transaction syntax used to look:<\/span><\/p>\n\n\n\n

\"\"<\/p>\n\n\n\n

Lately, it seems like handling transactions inside lambda functions is becoming the norm for NoSQL databases.<\/span><\/p>\n\n\n\n

For those who were expecting it to be similar to the relational syntax for transactions (e.g. BEGIN\/COMMIT\/ROLLBACK SQL commands), keep reading: you can also run transactions through N1QL! Now, let\u2019s try to understand what is happening under the hood.<\/span><\/p>\n\n\n\n

\u00a0<\/p>\n\n\n\n

Couchbase Architecture Review<\/span><\/h3>\n\n\n\n

For those unfamiliar with Couchbase\u2019s architecture, I need to quickly explain 4 important concepts before going any further:<\/span><\/p>\n\n\n\n