Server error during processing: ["web request failed"...]

Couchbase Server
1

Hi,

We are running a three node Couchbase server 2.0.1 cluster in our dev/integration environment, to which multiple instances of our java app connect using the Java SDK.
We are seeing the following error, over and over, in the couchbase server log:

Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 371 times)

We are also seeing timeout errors in our application:

2013-07-02 14:22:56.686 WARN com.couchbase.client.vbucket.ConfigurationProviderHTTP:  Connection problems with URI http://rnd-couchbase1.fiftyone.com:8091/pools ...skipping
java.io.IOException: Timed out while reading configuration over HTTP
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readToString(ConfigurationProviderHTTP.java:442)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readPools(ConfigurationProviderHTTP.java:230)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:148)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:320)
        at com.couchbase.client.CouchbaseConnectionFactory$Resubscriber.run(CouchbaseConnectionFactory.java:406)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:887)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
        at jrockit.net.SocketNativeIO.readBytesPinned(Native Method)
        at jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:32)
        at java.net.SocketInputStream.socketRead0(SocketInputStream.java)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:224)
        at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:148)
        at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)
        at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:377)
        at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readToString(ConfigurationProviderHTTP.java:417)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.readPools(ConfigurationProviderHTTP.java:230)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:147)
        at com.couchbase.client.vbucket.ConfigurationProviderHTTP.subscribe(ConfigurationProviderHTTP.java:318)
        at com.couchbase.client.CouchbaseConnectionFactory$Resubscriber.run(CouchbaseConnectionFactory.java:406)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        ... 1 more

This is really concerning to us as we have chosen Couchbase to increase our apps stability (which started suffering with Oracle as we grow).

Any help in diagnosing this issue will be appreciated.

2

Hello,

Can you elaborate…

You application is working correctly (so your client is connected to the cluster), and sometimes you see this error?

Or you cannot connect at all to the cluster?

Regards
Tug
@tgrall

3

Tug,
Sorry, just saw you comment…
We are having intermittent problems where our java client’s get/set operations fail with timeouts.

4

Hello,

In case you have still have the problem…

First of all, be sure that you are using a singleton to connect to each bucket. As you probably know all the “connection pooling” is done in the Couchbase Java client itself. For example we create the connection in an ServletContextListener in our sample application: https://github.com/couchbaselabs/beersample-java/blob/master/src/main/java/com/couchbase/beersample/ConnectionManager.java

Quiet often time out are related to under-sized cluster can you give us some information on the number of operation per seconds, documents and number of sizing of you cluster?

Also the most obvious one… can you check that the network. :wink:

PS: Sorry for the delay, I did not receive the notification and missed your comment.

Regards
Tug
@tgrall