Missing keys/replicas
Either there is something I have misunderstood about membase or a bug can be provoced out in the open.
I have a website backed by 2 membase nodes. (sep. hardware, windows 2008)
I use membase to store web session data and persistent info on how my analysis databases are up to data.
Membase gives the users a parallel experience (thru my webservices which have no state), and the system have the ability to loadbalance requests between multiple webservers.
I also loadbalance the analysis load on my databases via persistent info in mambase.
So I need membase to be reliably persistent.
I use 1 day as timeout for the keys, and updates this timeout on get operations too. (so I get a running window of timeout)
Since i've noticed the fact that keys suddenly are missing from membase, I have been investigating every aspect of my code related to membase.
I use the enyim v2.12 client. (latest)
In writing momment i have the following item stats, that worry me
Active items Replica Items membase1.prod.local.netminers.dk 339 310 membase2.prod.local.netminers.dk 333 322
So i wonder...
If Server 1 goes down now, would I loose 339-322=17 keys?
Right now I have doubts about if membase can work for me.
When I reset my keys I can see the replica works fine (expected) for a couple of minuttes, then suddenly replicas starts missing.
Enyim client settings.
I use the following client config on all systems using membase:
I have installed the membase servers using the guide provided for cloud install, since I wanted to use the name of the server and not a fixed ip. (for easier maintenance and flexibility)
Mayby I'm using membase wrong, or expect another behaviour than its designed for.
Hope anyone can help me out with some advice.
Hi Perry
First, thank you for answering my question.
You really got me on the inside before I was able to report my latest findings - in your favor
It turned out that the missing keys problem was a basic concurrency bug on my behalf!
I did totally expect that since lost of sites succesfully use membase/memcached.
The only reason I chose to write your forum was my concern with the replication stats.
I thought that maybe there were some relation between the replication stats and my missing keys.
I totally trust your replication scheme when you tell me that the system needs to stabilize before the stats are correct
So I do not want to waste anymore of your time with my previous concerns!
I'm a true membase believer now, and will spread the word around :)
Thanks for the explanation!
Michael
Hi Michael, your desire for Membase to be reliable and persistent is certainly acceptable and we have a number of customers using it for just that.
To answer your initial question, Membase does not use a traditional "master-slave" replication. With 2 nodes, you have basically that, but it's better to think of each vbucket (there are 1024) being replicated individually and all of that spread out throughout the cluster. It starts to make more sense when you think of 3 or more nodes...node1 would actually be replicated to both 2 and 3 instead of just one of them. Does that make sense?
Now, in order to figure out what's going on, we'll want to get a reliable case to reproduce. Depending on how fast you are changing things, I would expect the statistics to possibly be a bit out of date, but once things settle down, they should be accurate.
First off, what version of Membase are you running?
-We want to make sure that the keys are getting set reliably in the first place. It sounds like you've already got that since you're able to access the keys right after putting them in, but a number of customers have run into situations where a set/store actually returns an error which they don't check and therefore assume that they was stored successfully...Let's turn on logging on the client machines to see if there are any errors or whether the keys are just plain missing: https://github.com/enyim/EnyimMemcached/wiki/Configure-Logging
-Let me confirm one of your statements...you put keys in, reliably get them back, and then just a few minutes later start to be missing the same keys? What happens if you manually telnet to Membase and do a 'get :11210 all" and look for "ep_expired". if that number is increasing, and correlates with the same time you are requesting keys, then it would appear that they are being expired and perhaps there is something incorrect with the way you are setting the expiration itself.
Lets start with that and see what we can find.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!