|
I had fixed this with couchbase buckets and we test for this in SDKQE. Is this possibly isolated to memcached buckets.
|
|
Actually, this has been discovered while using Couchbase buckets. Let's chat about this, but I think thats a different issue and not related to it.
|
Sure, the solution previously was to have that threshold if we were getting unexpected failures. Once we pass that threshold, we'd try to re-subscribe.
I'm not opposed to a heartbeat, but I don't think the HTTP HEAD is good, since the mochiweb erlang implementation is effectively the same as a GET. I'll reach you and chat through it.
|
Okay after talking this through with Matt, I reran the script to see if the proposed solution (increasing ops/s) fixed the problem.
Interestingly, it turns out when doing set/get's it never hits our anticipated codepath but instead the get operations just time out, the threshold never gets increased and nothing happens. This is something we need to reinvestigate.
|
|
Just try to do this with a simple java script (i.e. connect and query the view)
|
This issue persists when the view retrieval is ran from standalone java program as well.
I tried this on both the VMs - 10.3.3.203, 10.3.3.209.
|
|
Its working fine for server deployed on localhost.
|
|
Michael: found out today that this is a large issue for SDKQE. Can you have a quick look at this in the next day? You may find the underlying issue.
|
|
Please pass me the script as commented and then (or if you can't) please assign it back to me! Thanks
|
According to the posted logs, it looks like debugging was not turned on. Can you please run this again with debugging turned on? To get the full logs to STDOUT, use this before initializing the CouchbaseClient in The App:
// Tell spy to use the SunLogger
Properties systemProperties = System.getProperties();
systemProperties.put("net.spy.log.LoggerImpl", "net.spy.memcached.compat.log.SunLogger");
System.setProperties(systemProperties);
Logger.getLogger("com.couchbase.client").setLevel(Level.FINEST);
//get the top Logger
Logger topLogger = java.util.logging.Logger.getLogger("");
// Handler for console (reuse it if it already exists)
Handler consoleHandler = null;
//see if there is already a console handler
for (Handler handler : topLogger.getHandlers()) {
if (handler instanceof ConsoleHandler) {
//found the console handler
consoleHandler = handler;
break;
}
}
if (consoleHandler == null) {
//there was no console handler found, create a new one
consoleHandler = new ConsoleHandler();
topLogger.addHandler(consoleHandler);
}
//set the console handler to fine:
consoleHandler.setLevel(java.util.logging.Level.FINEST);
Would be great if we can get all the output so we can investigate where the timeouts come from. I'm sure with a debug log it will be much easier. Thanks!
|
So this bug isn't such an "unknown" anymore, and has exposed itself in 1.1.0 as well as 1.1.1 and isn't limited to particular clusters - (it just seems that some clusters are more likely than others to trigger this bug).
However this still needs a lot of care and analysis
|
|
This is "kinda" blocker.
|
failover-debug.args
fo-dbg.log.txt