Couchbase extension issues
I've run into a few issues in my production environment using the Couchbase extension that have me scratching my head pretty good. I'm running a cluster of 20 Centos 5.5 servers with Membase 1.7.1.1 installed. Connecting to them are 45 app servers with the Couchbase PHP extension, version 1.0.1. We went live with the application that's running on all this today and say a large amount of the following error:
Error: Couchbase::__construct(): Failed to connect libcouchbase to server: Connection failure
We saw the most errors when we had around 2500 ops/sec to the cluster. My initial thought is that I'm running out of connections. In the space of about 5 hours, one Membase server had a total of 10 million connections to it. The general flow of the program is we do
$membase = new Couchbase('my.membaseServer.com:8091', 'username', 'password', 'default');
Then from there we normally do a few gets and sets and then the script finishes execution. However, this happens quite a bit. The number of errors increased as our traffic increased and then the number of errors decreased as the traffic decreased. At this point, though, I'm not sure where to start looking. A couple things we've done is we took out the bucket specification in the Couchbase constructor call and we took out a call to getStats() that we were using to verify the connection. I looked into updating the Couchbase extension to the latest version, 1.0.4, since I saw in the release notes and various other forum posts that a timeout issue had been fixed as well as a few other things. But when I updated the extension on my test server I received a 'Couchbase::get(): unknown payload type' error, which had me puzzled too since the last time I saw that was when I switched from using the Memcached extension and moxi client to the Couchbase extension.
So at this point I'm pretty confused and not really sure where to start looking. Anyone have any insight into things I can look for to trouble shoot this error to get rid of those pesky failed to connect errors?
Thanks!
Hi,
This isn't really a tested combination, though it should work. The PHP client was tested with Couchbase Server 1.8.0 and has been re-tested with 1.8.1.
As to the error you're running into, it sounds like it's possible that you could be exhausting the number of connections on the server. One feature of the PHP client is to use persistent connections (there were some issues earlier, but they've been resolved in the 1.0.5 release). It may make sense to try persistent connections which will not only make things faster, but will also reduce the number of connections to one per worker process (I'm assuming Apache HTTPD in this case) and the requests will queue waiting for this shared persistent connection I do believe. You may want to try this on one of your app servers first.
If you believe there are possibly more than 10k concurrent requests, this may well help. It should provide a performance boost too.
As far as getting additional data, a "Connection failure" can be raised either when bootstrapping (the HTTP requests when determining the cluster layout) or when trying to connect to a node for data purposes and/or sending an operation and finding a node to have failed.
The first thing to do would be to check the server logs to see if there's anything in there that indicates we're hitting the limit on connections. I believe that'd be logged, but I'm not certain with 1.7.1.1. It would be in the stats though, so you could use mbstats to see how many connections are occuring. There could also be a limit to the number of connections.
The other thing you could do is packet capture ports 8091 and 11210. Look for any FIN from 111210 to the client or for any HTTP responses that are below 300 on port 8091. This is certainly lower level debugging that you shouldn't have to do-- hopefully the logs will tell you more already but again-- I'm not sure about version 1.7.1.1.
Thanks,
Matt