Another error that popped up after I migrated a service to use java sdk 3.0:
We do intensive usage of atomic counters. I have a class that does get, getAndTouch and increment.
After a while running the service (X100 request per seconds) I’m getting Timeout exception with reason Endpoint Not Writable.
The exact logs are:
[com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“GetRequest”,“server_us”:0,“last_local_id”:“D6021CA500000001/0000000016FCC870”,“last_local_address”:“10.109.35.160:32284”,“last_remote_address”:“10.108.0.96:11207”,“last_dispatch_us”:815997,“last_operation_id”:“0x10870b3”,“total_us”:1926025},{“operation_name”:“GetRequest”,“server_us”:0,“last_local_id”:“D6021CA500000001/0000000016FCC870”,“last_local_address”:“10.109.35.160:32284”,“last_remote_address”:“10.108.0.96:11207”,“last_dispatch_us”:815991,“last_operation_id”:“0x10870b2”,“total_us”:1926022}
and then:
[com.couchbase.config][BucketConfigRefreshFailedEvent] Reason: INDIVIDUAL_REQUEST_FAILED, Type: KV, Cause: com.couchbase.client.core.error.RequestCanceledException: CarrierBucketConfigRequest {“cancelled”:true,“completed”:true,“coreId”:“0xd6021ca500000001”,“idempotent”:true,“reason”:“NO_MORE_RETRIES (ENDPOINT_NOT_WRITABLE)”,“requestId”:17455328,“requestType”:“CarrierBucketConfigRequest”,“retried”:0,“service”:{“bucket”:“ad-stats”,“collection”:"_default",“opaque”:“0x10a439b”,“scope”:"_default",“target”:“10.108.0.96”,“type”:“kv”},“timeoutMs”:2500} {“coreId”:“0xd6021ca500000001”}
from this point all requests fail
Note that I tried with several kv circuit breakers definition, disable circuit breaker at all, with several retry strategies (best effort and fail fast)
Previous code with sdk 2 worked fine.
My question is what can cause this issue and how can I avoid it?
Thanks,
Asher