Retries occurred each 10 request to db

Java SDK
, , ,
1

Hi!

I’m using Java 11 with JDK version 3.4.1

The infrastructure looks like API Gateway + Lambda + Couchbase server on an EC2 instance.

Each 6 - 10 request to Lambda occurred retries with “SERVICE_NOT_AVAILABLE” and “NODE_NOT_AVAILABLE” reasons

1Capture
1Capture1589×402 41.5 KB

2Capture
2Capture1300×667 165 KB

3Capture
3Capture1319×378 70.4 KB

And then it appears as WARN that “Initializing the global config failed: UNKNOWN”

Code:

Cluster environment configuration: 

        Duration timeoutDuration = Duration.ofSeconds(30);

        Consumer<TimeoutConfig.Builder> timeoutConfig = item -> item
                .kvDurableTimeout(timeoutDuration)
                .kvTimeout(timeoutDuration)
                .queryTimeout(timeoutDuration)
                .searchTimeout(timeoutDuration)
                .connectTimeout(timeoutDuration)
                .analyticsTimeout(timeoutDuration)
                .disconnectTimeout(timeoutDuration);

Request to Couchbase:

 String queryAsString = "...";
 cluster.query(queryAsString);

Will appreciate your help!

2

It looks like the lambda does not have access to the cluster. Please make sure you’ve exposed all relevant ports (Couchbase Server Ports | Couchbase Docs) and can contact them externally.

3

I googled that it can be a possible cause and then exposed all ports
Here is the screenshot of inbound rules for lambda

image
image1026×398 14.7 KB

4

Those are only the non-ssl ports. For ssl, the ssl ports also need to be accessible.

You can use curl to check that the ports are accessible - timeout means they are not exposed, anything else - include errors is exposed (curl will give errors when a non-http/s port is accessed).

5

I’m not using ssl connection way and have a Couchbase community edition.

Here is the line of connection to the cluster

 cluster = Cluster.connect("connection string", clusterOptions);

Is not opened SSL ports are still relevant for my case?

6

Presumably “connection string” is not your actual connection string :grinning:

No, the SSL ports shouldn’t be used if you’re not using SSL.

But this is still likely to be a basic connectivity/ports issue. You’ve shown the AWS lambda inbound rules, but where is the cluster hosted - what about the inbound & outbound rules configured there? Can you connect to the cluster from your laptop? sdk-doctor (GitHub - couchbaselabs/sdk-doctor: Application-server-side cluster connection diagnostics.) can be useful for identifying connectivity problems.

7

Cluster hosted on EC2 under the same account and region as lambda which uses Couchbase. Cluster has exactly the same security group as lambda (you can check the screenshot with lambda inbound rules above)

Thanks for helping, I will try to use sdk docktor

8

Tested cluster connection with sdk-doctor and didn’t notice any errors or issues:

image
image939×299 14.4 KB

Any ideas about possible reasons and ways how to diagnose this problem?

Will appreciate your help!

9

Each 6 - 10 request to Lambda occurred retries with

But they eventually succeed? If they fail can you show the stack trace?

The messages - especially the DEBUG message that says “ignored on purpose” is a not a fatal error. To get the configuration, couchbase will try the management service if it fails to get the config from the kv service. And it will also retry. The operation may timeout, but won’t fail when couchbase temporarily does not have a connection.

Can you show the connection string you are using? Can you show the SDK Doctor command and all the output from it?

I’m using Java 11 with JDK version 3.4.1

Without any particular change in mind, the newer versions tend to work better. Could you try 3.4.6?