The remote side disconnected the endpoint unexpectedly - warn in Java sdk 3.0 + couchbase server 6.5.1

[herat_acharya]

I am using SDK 3.2.2 with server version 6.6.1 … I also get this UnexpectedEndpointDisconnectedEvent
Because of which the queries get interrupted and i get InterruptedException. Is there a way too find out frequent disconnections with couchbase what should i enable? Also what are the ways to mitigate that … We upgraded from 2.7.20 to 3.2.2 Java client and in older version we didnt get that.

We are struggling to find a solution and its been going on for some time… Any help is appreciated.

@ingenthr @daschl

[daschl]

@herat_acharya please follow up in the thread you already created based on my input (i.e. providing the logs)

[launchpad]

This doens’t seem to work with spring-data-couchbase 5.0.1. I had to change it via code:

@Override
  protected void configureEnvironment(ClusterEnvironment.Builder builder) {
    super.configureEnvironment(builder);

    builder.ioConfig().idleHttpConnectionTimeout(Duration.ofMillis(4000));
  }

It defaults to 4.5 seconds already:

  /**
   * By default, idle pooled endpoints are cleaned up every 4.5 seconds so that
   * they do not clash with idle socket close attempts by the server.
   */
  public static final Duration DEFAULT_IDLE_TIME = Duration.ofMillis(4500);

[launchpad]

Hey, @ingenthr from what I understand if the idleHttpConnectionTimeout is higher than the query service default idle time then this will arise.

If that’s not the case then you mentioned to look at the server logs can you please point me to which server logs would be helpful here as I didn’t find any errors in the query.log file on the query nodes.

[mreiche]

@launchpad - I maintain spring-data-couchbase. Please note that this is a very old thread.

It would be a good idea to use the latest spring-data-couchbase which is 5.1.4. It uses a recent Couchbase Java SDK - I believe it had an httpIdleTimeout of 1 second.

spring.couchbase.env.io.idle-http-connection-timeout=4s

spring-data-couchbase had not supported configuration via properties after Spring Data Couchbase 3.x, so yes, to change couchbase settings, your implementation of AbstractCouchbaseConfig will need to make thise changes.

[launchpad]

Hey @mreiche thanks for the suggestion and confirming the configuration style.

Reducing the idleHttpTimeout to 1s from 4.5s has definitely reduced the frequency of the WARN logs but it still happens once in a while.

Do you have any other suggestions?

Also, from what I understand if the idleHttpConnectionTimeout is higher than the query service default idle time then this issue will arise. Where can I see this configuration of query service?

[mreiche]

isn’t query service default idle time in fact idleHttpConnectionTimeout ?

Can you post the message you are seeing?

• Mike

[launchpad]

@Han_Chris1

WARN [com.cou.endpoint] (cb-events) [com.couchbase.endpoint][UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly {“circuitBreaker”:“DISABLED”,“coreId”:“0x1ec6629500000001”,“local”:“127.0.0.1:57170”,“remote”:“localhost:8093”,“type”:“QUERY”}

It’s exactly this @mreiche.

[mreiche]

@ingenthr

Note for anyone reading this thread, the WARNing in the logs with:

@Poltar

[UnexpectedEndpointDisconnectedEvent] The remote side disconnected the endpoint unexpectedly

Is a generic message. The specific cause can’t be determined from the SDK log. It could be anything from the query process crashing to the remote computer crashing to some kind of network change that closes the TCP connection. To understand more specifically why, the server side logs would need to be examined.

There was one cause which was a change in the query service default idle time I describe above.

If changing the idleHttpConnectionTimeout doesn’t have an impact, then it’s likely the cause of the server side dropping the connection is something else. The SDK won’t know that cause.

Ok - so that’s what ingenthr replied to earlier. If the http idle timeout is set to 4 seconds (or 4.5 seconds) and the server http idle timeout is 5 seconds - it should not be from the server closing the connections as “idle” as the SDK would have closed its side first. There may be some additional information in the query server log. Or it could be something in between dropping the connection - like a router. As long as it doesn’t occur during a query - it shouldn’t cause any problem as the SDK automatically reconnects.