Hot upgrading couchbase servers - failed request using couchbase net client 3.1.1

.NET SDK
1

Hi I am using the latest client version 3.1.1.

I decided to hot upgrade couchbase server from 6.6.0 to 6.6.1.
I have 3 server node cluster, first i add one new node and everything was ok, then I upgraded node by node with failover mode. When I removed (failover node) i was getting this type of errors

Couchbase error message: The operation 3414384/3414384 timed out after 00:00:30. It was retried 34 times using Couchbase.Core.Retry.BestEffortRetryStrategy. stack: at Couchbase.Core.Retry.RetryOrchestrator.ThrowTimeoutException(IOperation operation, Nullable`1 timeout)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationToken token)
at Couchbase.KeyValue.CouchbaseCollection.GetAsync(String id, GetOptions options)
at Shared.CouchbaseClient.CouchbaseClient.GetWithCasAsync[T](String key, Boolean returnObject) in /home/ubuntu/workspace/4O_Backend/src/Lib.Shared/CouchbaseClient/CouchbaseClient.cs:line 475 bucket name: Failed to get document by key: objRevs:ABbvhs5TeJHsEv06QsQOIZlpapg0T

When I upgraded all nodes i got this type of errors

Couchbase error message: Cannot access a disposed object.
Object name: ‘DataFlowConnectionPool’. stack: at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.<>c.b__120_0(IOperation op2, Object state, CancellationToken effectiveToken)
at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationToken token)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationToken token)
at Couchbase.KeyValue.CouchbaseCollection.GetAsync(String id, GetOptions options)
at Shared.CouchbaseClient.CouchbaseClient.GetWithCasAsync[T](String key, Boolean returnObject) in /home/ubuntu/workspace/4O_Backend/src/Lib.Shared/CouchbaseClient/CouchbaseClient.cs:line 475 bucket name: Failed to get document by key: objChg:ABbvd6kOlhoFcTk-nPbfEfi_IwA0T:1

With previous version i havent this problems when i was upgrading servers. I will downgrade version and let you know if old version (3.0.7) works

2

@luka -

Thanks for reporting, I created a ticket for tracking: https://issues.couchbase.com/browse/NCBC-2797

Please let us know how this goes!

-Jeff

3

Hi,

I tried with version 3.0.7 but it is better since the system recovered after few errors.

First we did graceful failover on one of the 4 nodes. As soon as CB started with failover, errors below start to appear…

Couchbase error message: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter ‘index’) stack: at System.Collections.Generic.List`1.get_Item(Int32 index)
at Couchbase.KeyValue.GetResult.ParseSpecs()
at Couchbase.KeyValue.GetResult.ContentAsT
at Shared.CouchbaseClient.CouchbaseClient.GetWithCasAsync[T](String key, Boolean returnObject) in /home/ubuntu/workspace/4O_Backend/src/Lib.Shared/CouchbaseClient/CouchbaseClient.cs:line 480 bucket name: Failed to get document by key: ui_uuser_18618493

…when failover was done, we had no more errors.

But then it came “Rebalance”. When we started that again errors “Index was out of range.” appeared. And we had that for a minute or two, even when rebalance completed.

And for the final we had few errors below…

Status code: ‘ServiceUnavailable’ Response body: ‘{“message”:“Couchbase DB is temporarily unavailable, please try again later. Bucket:Failed to get document by key: uai_ua_ui_user_16212112”,“errorCode”:“ServiceTempUnavailable”}’

…and then everything went back to normal.

1 Like
4

Thanks @luka for the feedback.

-Jeff