Data loss with Membase
I started looking into membase for one thing and while working on another problem have begun to wonder if membase might be suitable for it as well. I have a few concerns though that I haven't yet explored regarding my understanding of how membase works.
As I understand it when you set a value that operation is atomic and there are options to implement an optimistic offline lock type pattern. My question though relates to the potential for data loss. As I understand it when you set a value the value is set in memory only before the client is informed the value has been set. The data is persisted to disk in an asynchronous fashion and is also replicated I'm assuming in an asynchronous fashion.
Thus if I set a value and the client moves on thinking all is well if something happened to the cluster node the value was set on such as a server crash, process restart etc there is a potential the data may not have been written to disk or replicated to another node. The key would just disappear.
My use case involves using membase to store some semi-structured data that is referred to elsewhere in my application (our relational database among other places). My concern is that if a user is lead to believe something has been stored and given an id to refer to it and it is referred to elsewhere in the system but then can't actually be found because of the scenario above that would not be good.
Now in talking about the scenario with my team we agreed if the occurrence of this was really low we could live with it, the data isn't like bank account data or anything like that but if it happened frequently we would be in trouble. I'm thinking it would be an exceptional circumstance but without any actual membase experience I'm not sure.
What are the best practices if you need to take a cluster node down for maintenance? How do you ensure all data is flushed to disk and replicas before shutting down? Do you usually just activate all the replicas on other nodes and then the node going down will be "idle"?
I'm sure my scenario isn't unique and I'm curious how other people are dealing with things or if there is something I'm misunderstanding which eliminates or greatly reduces the risk I see. How durable is data in membase in ACID terms?
Thanks,
Bryan
Additionally Bryan, our replication is one of the very first things that happens to an item after it is modified and is designed to happen extremely quickly.
The statistics are available to determine whether you have a backlog of replicated items and so you can understand your "exposure" to dataloss if a node fails. Ideally, and all we've seen, this is always very low and the chance of dataloss in a Membase system is extremely rare.
We are "ACID" on a per-item basis with asynchronous durability. There are roadmap items to allow the administrator to make certain datasets "synchronous" but you'll obviously have to deal with the performance tradeoffs.
Hope that helps
Perry
Bryan,
You are right. By the time the asynch disk write fails, the application has already been notified that the "write" succeeded. Disk writes are on a best efforts basis and the application is taking some level of risk that the data may actually not be durable.
But it should be noted that if the disk write failure is transient (vs. a disk crash), it will be retried by the server. There is a chance of data loss but it is limited to the chance of a disk crash or power failure while the data is being written to the disk.
The best practice of taking a node out for maintenance (in the current release) is to 'remove' a node. Note that 'remove' just marks the node for removal. After 'remove', rebalance the cluster to completely remove the node and redistribute its resources (the active and the replica data) to the other servers in the cluster.
Bhawana
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!