vBuckets
Reading a more technical explanation of how vBuckets work, each vBucket is defined in one of the following states: Active, Pending, Replica and Dead. (Source: http://dustin.github.com/2010/06/29/memcached-vbuckets.html)
Quoting the document, "A replica vbucket is similar to a dead vbucket in that from a normal client’s perspective. That is, all requests are refused, but replication commands are allowed."
Lets say we have a simple two node cluster, with 1 replica copy vBuckets. We can assume, that each vBucket has an Active state on one server, and a Replica state on the other server (once data in bucket is replicated fully). The above description makes me believe, that the Replica copy is never used, even for reading. Say you are connected to server B, which has a Replica copy of the vBucket you're trying to read a specific key from; the connection will always be made to Server A for the active bucket, correct?
Is there a possibility to control the behavior in such a way, that a Replica copy could also be used for reading (read only) key values (which I assume, could be stale because of replication lag). This could be defined on the bucket level (just like number of replicas).
I see some benefits in this:
1) Auto-"failover" key reads in the time before it takes the cluster to know a node went offline. Writes are still blocking until the node has been actually failed-over.
2) Reduced network traffic between couchbase nodes since reads from Replicas may be satisfied on the server you're connecting to (assuming, the server has a Replica copy of the vBucket you need);
A slightly improved design of vBuckets could provide a "floating-master" (Active) vBuckets, which would migrate between servers if write-blocking should occur on the node which holds the currently-active vBucket (due to the node being taken offline for example). When the node would re-join the cluster, the buckets would be resolved from Pending state into Replication (to copy updated data in the cluster) and then eventually, like with rebalance operation, the "active" vBucket states and data could be distributed across the nodes.
Is this something you're researching, or has the architecture already changed in a significant way since the original vBuckets document almost 2 years ago? How relevant is this today with CB 2.0?
Thanks,
- Tit
That's awesome! ;) But I assume, that writes would still block in a scenario, where some data is unreachable?
In a master-master scenario, where the machines would on occasion be unreachable between themselves, but would each have clients connecting and writing to them, would it be possible to fail over in such a way, that a replica copy would become active on each of the servers, allow writes to it, and when they are visible to each other again, resolve inconsistencies between buckets and keys across the server (sort of like rebalance does).
By using a combination of vector clocks/deltas, on vBuckets as well as on memcache keys stored in them, resolving conflicts would be made possible. I'm guessing some kind of vector clock already exists, to keep track what data was replicated from the active vBucket into it's replica copies?
>> But I assume, that writes would still block in a scenario, where some data is unreachable?
If a server is unavailable then writes will block (and most likely timeout in the client). What you would need to do is manually fail over the server to activate the replicas. Recent versions of Couchbase also allow auto-failover which can recognize a down node in around 30 seconds.
>> master-master scenario
Auto failover allows you to do this, but not in the way you describe. One thing to remember is that you must have a reasonable threshold for the minimum time a server can be unresponsive before you consider it down so writes will block for a some period of time. One thing that might not be optimal here is that we will move all of the data back to the failed node once it comes back and we don't do conflict resolution between the new master and the failed node that was added back. This might be something we can improve in the future.
>> I'm guessing some kind of vector clock already exists, to keep track what data was replicated from the active vBucket into it's replica copies?
We only write to the master node so the master always has the latest copy of the data. As a result conflict resolution is not needed.
>> Say you are connected to server B, which has a Replica copy of the vBucket you're trying to read a specific key from; the connection will always be made to Server A for the active bucket, correct?
Yes. If you were to ask server B for the key it would say it didn't have it. Couchbase clients either uses a proxy server on each node to redirect traffic to the correct server or they get a vbucket map from one of the servers in the cluster that can be used to figure out which server has the active vbucket for that key.
>> Is there a possibility to control the behavior in such a way, that a Replica copy could also be used for reading (read only) key values ?
We recently added a read from replica command that will allow you to do this.
>> Reading of replicas could be beneficial when a node is down
This is exactly why we added the read from replica command
>> Reduced network traffic between couchbase nodes
While you could use replica reads to reduce network traffic between servers I think the better approach for the client is to take advantage of the vbucket maps that are provided by the server since replica read means that you can read stale data.
>> Is this something you're researching, or has the architecture already changed in a significant way since the original vBuckets document almost 2 years ago? How relevant is this today with CB 2.0?
The vbucket design is still intact in all Membase and Couchbase products including Couchbase 2.0.