A major operation in a cluster of membase servers is called Rebalancing. A membase system administrator may choose to initiate a Rebalance because new servers might have been added, old servers need to be decommissioned and need to be removed, etc. An underlying part of Rebalancing is the controlled migration of vbuckets (and the items in those migrating vbuckets) from one membase server to another.

There's a window of time, given the distributed nature of membase servers and clients, where vbuckets ownership may have changed and migrated from one server to another server, but your client library has not heard the news yet. So, your client library would be trying to talk to the 'wrong' or outdated server for a given item, since your client library is operating with an out-of-date vbucket-to-server map.

Below is a walk through of this situation in more detail and how to handle this case:

Before the Rebalance starts, any existing, connected clients should be operating with the cluster's pre-Rebalance vbucket-to-server map.

As soon as the Rebalance starts, membase will "broadcast" (via the streaming REST/JSON channels) a slightly updated vbucket-to-server map message. The assignment of vbuckets to servers doesn't really change at this point at the start of the Rebalance, but the serverList of all the servers in the membase cluster does change. That is, vbuckets have not yet moved (or are just starting to move), but now your client library knows the addresses of any new membase servers that are now part of the cluster. Knowing all the servers in the cluster (including all the newly added servers) is important, as you'll soon see.

At this point, the membase cluster will be busy migrating vbuckets from one server to another.

Concurrently, your client library will be trying to do item data operations (Get/Set/Delete's) using its pre-Rebalance vbucket-to-server map. However, some vbuckets might have been migrated to a new server already. In this case, the server your client library was trying to use will return a NOT_MY_VBUCKET error response (as the server knows the vbucketId which your client library encoded into the request).

Your client library should handle that NOT_MY_VBUCKET error response by retrying the request against another server in the cluster. The retry, of course, might fail with another NOT_MY_VBUCKET error response, in which your client library should keep probing against another server in the cluster.

Eventually, one server will respond with success, and your client library has then independently discovered the new, correct owner of that vbucketId. Your client library should record that knowledge in its vbucket-server-map(s) for use in future operations time.

An implementation of this can be seen in the libvbucket API vbucket_found_incorrect_master().

At the end of the Rebalance, the membase cluster will notify streaming REST/JSON clients, finally, with a new vbucket-to-server map. This can be handled by your client library like any other vbucket-to-server map update message. However, in the meantime, your client library didn't require granular map updates during the Rebalancing, but found the correct vbucket owners on its own.