Data loss with Membase
I started looking into membase for one thing and while working on another problem have begun to wonder if membase might be suitable for it as well. I have a few concerns though that I haven't yet explored regarding my understanding of how membase works.
As I understand it when you set a value that operation is atomic and there are options to implement an optimistic offline lock type pattern. My question though relates to the potential for data loss. As I understand it when you set a value the value is set in memory only before the client is informed the value has been set. The data is persisted to disk in an asynchronous fashion and is also replicated I'm assuming in an asynchronous fashion.
Thus if I set a value and the client moves on thinking all is well if something happened to the cluster node the value was set on such as a server crash, process restart etc there is a potential the data may not have been written to disk or replicated to another node. The key would just disappear.
My use case involves using membase to store some semi-structured data that is referred to elsewhere in my application (our relational database among other places). My concern is that if a user is lead to believe something has been stored and given an id to refer to it and it is referred to elsewhere in the system but then can't actually be found because of the scenario above that would not be good.
Now in talking about the scenario with my team we agreed if the occurrence of this was really low we could live with it, the data isn't like bank account data or anything like that but if it happened frequently we would be in trouble. I'm thinking it would be an exceptional circumstance but without any actual membase experience I'm not sure.
What are the best practices if you need to take a cluster node down for maintenance? How do you ensure all data is flushed to disk and replicas before shutting down? Do you usually just activate all the replicas on other nodes and then the node going down will be "idle"?
I'm sure my scenario isn't unique and I'm curious how other people are dealing with things or if there is something I'm misunderstanding which eliminates or greatly reduces the risk I see. How durable is data in membase in ACID terms?