Couchbase library giving exceptions during load

Hello,

We are using couchbase v2.1.1 community edition for storing about 4 Million documents.

We access it using Couchbase Java library v1.2.2

During high load of around 6000 requests / second, Couchbase library is throwing below mentioned exceptions;

OperationTimeoutException,
CheckedOperationTimeoutException,
CancellationException,
ExecutionException

We are using couchbase library methods which return OperationFuture objects, however not waiting on Future.get, (in a non-blocking way)

Pseudo code:

{
com.couchbase.client.CouchbaseClient.set(key, value, PersistTo.ZERO, ReplicateTo.ZERO);
return;
}

We have used some of the below configuration to initialize the Couchbase client using Couchbase builder;

opQueueMaxBlockTime = 2000
shouldOptimize = true
obsPollMax = 2
obsPollInterval = 1000
viewTimeout = 3000
opTimeout = 1500

Hardware configuration:

Couchbase cluster:
Nodes = 4
CPU = 4 x 2.6 GHz
System RAM = 8 GB
Bucket RAM = 6 GB

Could anyone please provide any insight into this?

Thanks..

1 Answer

« Back to question.

Hi,

thanks for posting your question here on the Q&A forums. With regard to your problem, your description is very vague.

While not blocking on the futures is okay, you should do it at some point and inspect the result of the operation and see if it was successful or not. If it wasn't you can inspect the status of the future and see what's going on.

Also, according to your description you are not blocking, but those timeout exceptions are only thrown on sync operations (or when you block on the future with .get()).

Here are some general tips that may help you remedy this issue:

- Monitor the server stats closely on things indicating issues (disk fetches for example, leading to longer responses and probably timeouts)
- Check the network for outliers (long running pings to see if you get spikes)
- See if you find more infos in the error logs on the client side, possibly turning on debug logging.

To debug this issue further, we at least needs some client side logs and some sample code to run. Timeouts mostly indicate some temporary issue indicating contention of a resource and overload. How often are you seeing those timeouts?

Hi Michael, thank you for your reply. I will check and get back with more info as possible.