Endpoint_not_writable

[Adar]

I use Couchbase Community Edition 6.6.0, with a total of 8 Node nodes
A problem occurred recently, the error message is attached below
But at this time, my service CPU is only about 75%, which is not a high-stress state. How can I check whether one of my nodes is broken?

com.couchbase.client.core.error.AmbiguousTimeoutException: UpsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x26927ae300000001","idempotent":false,"lastChannelId":"26927AE300000001/00000000E4B7DF8E","lastDispatchedFrom":"*:43318","lastDispatchedTo":"*:11210","reason":"TIMEOUT","requestId":27721703,"requestType":"UpsertRequest","retried":1,"retryReasons":["<mark>ENDPOINT_NOT_WRITABLE</mark>"],"service":{"bucket":"default","collection":"_default","documentId":"user-3423344","opaque":"0x1a70288","scope":"_default","type":"kv","vbucket":467},"timeoutMs":10000,"timings":{"encodingMicros":624,"totalMicros":10009628}}

[mreiche]

Endpoint_not_writable usually means that the output channel is temporarily full. Specifying more channels (numKvConnections) can alleviate the situation. It’s odd that it only shoes one retry and the timeout is 10 seconds. The retry-backoff starts at 1 ms and increases up to 500ms, so I would expect to see tens of retries.

[Adar]

1.Output channel means couchbase output channel is full ?
2.I have 8 node, 20 game server connect, Each server use 4 numKvConnections, the num is too small?
3.The ten of retries is more suitable? why?

[mreiche]

See my previous post. If increasing numKvConnections does not resolve the issue, run with debug logging to get further diagnostics.

log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{ISO8601_OFFSET_DATE_TIME_HHCMM} %-5p [%c:%L] %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="com.couchbase.client.core.deps.io.netty" level="debug"/>
        <Root level="debug">
            <AppenderRef ref="Console"/>
        </Root>
    </Loggers>
</Configuration>

[mreiche]

@Adar

How can I check whether one of my nodes is broken?

Look at the log files for the node indicated by the “lastDispatchedTo” in the messages. But “ENDPOINT_NOT_WRITABLE” is usually not a problem with the node.