Capella cluster hangs

Hi,

I’m running into a strange situation with Capella Trial when the cluster suddenly stops responding to all incoming queries from both the SDK and the Query Workbench. Each of those queries eventually times out. On the other hand, they work perfectly fine with local Couchbase (Community Edition 7.0.2 build 6703).

The cluster does not respond even to simple queries such as select * from bucket limit 1.

I’ve tried to recreate the bucket, but now I cannot connect to the cluster because the Node.js SDK is throwing this error LCB_ERR_PROTOCOL_ERROR (1050): Data received on socket was not in the expected format","code":1050.

Do you have any suggestions on what could be the problem?

Thanks, Jakub

Hello Jakub,
Thank you for the decision to explore the Couchbase Capella, and are sorry to hear that.

One of the easiest ways to troubleshoot the SDK error is by using Couchbase SDK Doctor (link). Before doing that, please ensure to add your IP to the AllowedIP list and create corresponding Database Credentials.

It would very helpful if you could please share a Node.js SDK log file during the time when the timeout occurs.
Please see the corresponding docs here: Link

Thank you,
Sergei

Hi Jakub,

I was looking at the logs from this cluster for this issue you have reported.

I see there were some gateway timeouts that indicate the query service was not responding in a timely manner yesterday around 17:35 UTC time after a large query was run.

However since then I don’t see many issues, I also noticed all your buckets appear to have been deleted since.

I wonder are you still encountering this issue?

Regards,

Thanks for your replies.

Adam, so on Wednesday I tried to recreate the bucket - I deleted it, waited for it to disappear from the dashboard, and then created it again with the same name. I even updated the credentials because I was limiting their access only to the specific bucket.

The bucket (1) appears to exist in the dashboard:

Having that said I’m still having problems with connecting to the cluster, and therefore to both buckets. Here’s the output produced by SDK doctor:

|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|                                                                    |
|====================================================================|

Note: Diagnostics can only provide accurate results when your cluster
 is in a stable state.  Active rebalancing and other cluster configuration
 changes can cause the output of the doctor to be inconsistent or in the
 worst cases, completely incorrect.

12:20:13.730 INFO ▶ Parsing connection string `couchbases://cb.***.cloud.couchbase.com/***`
12:20:13.730 INFO ▶ Connection string was parsed as a potential DNS SRV record
12:20:13.790 INFO ▶ Connection string specifies to use secured connections
12:20:13.790 INFO ▶ Connection string identifies the following CCCP endpoints:
12:20:13.790 INFO ▶   1. ***.***.cloud.couchbase.com:11207
12:20:13.790 INFO ▶ Connection string identifies the following HTTP endpoints:
12:20:13.790 INFO ▶ Connection string specifies bucket `***`
12:20:13.842 INFO ▶ Performing DNS lookup for host `***.***.cloud.couchbase.com`
12:20:13.898 INFO ▶ Bootstrap host `***.***.cloud.couchbase.com` refers to a server with the address `18.219.110.10`
12:20:13.899 INFO ▶ Attempting to connect to cluster via CCCP
12:20:13.899 INFO ▶ Attempting to fetch config via cccp from `***.***.cloud.couchbase.com:11207`
12:20:14.597 ERRO ▶ Failed to fetch configuration via cccp from `***.***.cloud.couchbase.com:11207` (error: EOF)
12:20:14.597 INFO ▶ Not attempting HTTP (Terse), as the connection string does not support it
12:20:14.597 INFO ▶ Not attempting HTTP (Full), as the connection string does not support it
12:20:14.597 INFO ▶ Selected the following network type: 
12:20:14.597 ERRO ▶ All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible
12:20:14.597 INFO ▶ Diagnostics completed

Summary:
[ERRO] Failed to fetch configuration via cccp from `***.***.cloud.couchbase.com:11207` (error: EOF)
[ERRO] All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible

Found multiple issues, see listing above.

Thanks

@Jakub_Roman sorry about the delay - I’ll reach out to you directly to get some further details about your cluster so we can help you resolve this issue.