Clustering and rebalancing with Memcached
Hi,
We are planning to use memcached bucket with clustering setup. The application first tries to GET a key and then, if not found, would perform store operation. Suppose, there are two servers participating in a cluster. Consider following scenario.
1. Assume, both servers are up
2. There is a cache item (key-1) which is stored in Server-1
3. Now, Server-1 goes down
4. That means, key-1 would be now stored on Server-2
5. After sometime, server-1 comes up.
6. Key-1 would be now stored on server-1
7. At this point, both servers would have Key-1 stored. Is that correct?
8. Server-1 goes down again.
9. Now, if one performs GET operation for Key-1 then would GET succeed (Server-2)?
In clustering, there occurs rebalance when any node goes up or down. As far as I understand, rebalancing means, a key MUST be unique (across all participants in a cluster - if not unique, then it should be removed after rebalancing) at any point, after rebalance occurs. Is that inline?
One more general question, does rebalance of cached items take place with *memcached* buckets?
Still looking for inputs..... anybody?
oh.. its now over 3 weeks! no replies??????????
Yea these forums are pretty useless... so much for an active community... sorry I wouldn't be able to answer your questions tho.
I have to disagree enkrypt3d. We do try our best to answer all forum questions. Over the past 3-4 weeks we'd been prepping a new release as well. We do our best to turn around forum questions as fast as possible, but this one slipped.
Bhavnik: The scenario you list doesn't really indicate when you'd initiate failover (or use auto-failover) and rebalance.
The quick answer is that any time you initiate failover, replicas for the subset of the cluster not available are immediately promoted to available. This means if you repair a node and bring it back in to the cluster after a failover, you'll have to add it back and rebalance to bring that node back.
If no failover was ever initiated, then once the down node comes back online, all of the data which had been persisted will be available.
There are countless unanswered posts in here dude... and a majority of them are issues that many people have.
And its not just the past 3-4 weeks. Its been that way since August. We're all busy and we're here to try and find answers to an otherwise unanswerable issue.
We are definitely working on getting better about turning around answers more quickly.
ingenthr,
Thanks for your reply.
To be more precise, few more details....
Consider **Memcached** buckets are used.
As per my understanding, in clustering environment (Server-1, Server-2), membase/memcache (client?) would calculate a server on which a key (eg, Key-1) would be stored. This calculation happens according to some hash calculations of *available* servers at a time. This calculation forms an attachment of a cache key with one of the available servers.
For now, assume that Key-1 is always results into storage on Server-1, based on this hash calculation. After sometime, Server-1 goes down, and as a result, my application (Refer original scenario described by me in this post) will fail to GET operation on Key-1, and would initiate STORE operation for Key-1. All fine so far... Now, after few minutes Server-1 joins the cluster again (I don't care with whatever the method it brought back, manually/automatically/any-else). Here, while joining the cluster, rebalancing would occur. Right? (Question-1). Now, after joining, memcache client hash calculation would make the application to store Key-1 on Server-1.
Now, assume, my application looks for Key-1. Would GET operation for Key-1 succeed in app? (Question-2).
Considering above scenario, I'd expect Key-1 should be found from Server-1 again, if rebalancing guarantees the following protocol. "A CACHE KEY EXIST ONLY AND ONLY ON ONE SERVER (AT ANY MOMENT) ACROSS ALL SERVERS PARTICIPATING IN A CLUSTER, AFTER RE-BALACING"
Any thoughts?