Thread blocked on asynchronous bucket.get

Java SDK
1

Hello,

We are using java-client 2.6.2 and we are facing a strange issue in one of our UAT environment: a get operation is blocked .

Here the blocked part of our code :

JsonDocument doc = _cbBucket.async().get(key).retryWhen(retryWhenRuntimeException).toBlocking().firstOrDefault(null);

where :

 retryWhenRuntimeException = RetryBuilder.anyOf(java.util.concurrent.TimeoutException.class,RuntimeException.class)
    			.delay(Delay.exponential(TimeUnit.SECONDS, 64)).max(10)
    			.doOnRetry((t1, t2, t3, t4) -> _log.warn("Retrying due to a runtimeexception. retry number :{}:", t1, t2))
    			.build();

In the thread dump, we can see the following:

    "Thread-44" #243 daemon prio=5 os_prio=0 cpu=1471.97ms elapsed=63241.96s tid=0x00007fcc54b5f000 nid=0x1e1 waiting on condition  [0x00007fcd789a6000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.1/Native Method)
	- parking to wait for  <0x000000063f7d92b0> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.1/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.1/AbstractQueuedSynchronizer.java:885)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.1/AbstractQueuedSynchronizer.java:1039)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.1/AbstractQueuedSynchronizer.java:1345)
	at java.util.concurrent.CountDownLatch.await(java.base@11.0.1/CountDownLatch.java:232)
	at rx.internal.util.BlockingUtils.awaitForComplete(BlockingUtils.java:47)
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:460)
	at rx.observables.BlockingObservable.firstOrDefault(BlockingObservable.java:195)
	at xxx.CbStore.get(CbStore.java:75)

Any suggetsion please?

2

If the Java client doesn’t get a response from the server inside a specified time, it should raise a RequestTimeoutException. There shouldn’t be any way it can truly block.

Are you certain it’s blocked and not live-locked, e.g. maybe it’s repeatedly raising an exception and the retry logic you have there is retrying the operation indefinitely?

Wireshark could help shed some light on if this is happening. You can filter for “couchbase.key==‘my-key’”

3

Thanks grapham for your quick response
I have shared the retrypolicy code we are using, it is waiting for only few minutes at max and also it should log issue if there are but no trace.

4

Ah yes, I’d missed the .max(10) on the end there.
I’m sorry then, I’m not sure why this is happening. I can just say that it shouldn’t… It’s worth upgrading to the latest Java client though, 2.6.2 is getting on for a year old now.

5

@graham.pople I want to add that is issue is observed when doing intensive query to couchbase (many get operations).
I’m also seeing the following exceptions when the couchbase stop responding to my requests:

497064 [RxComputationScheduler-1] WARN com.couchbase.client.core.endpoint.AbstractGenericHandler - [/10.244.66.130:8093][QueryEndpoint]: Got error while consuming KeepAliveResponse.
java.util.concurrent.TimeoutException
	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber.onTimeout(OnSubscribeTimeoutTimedWithFallback.java:166)
	at rx.internal.operators.OnSubscribeTimeoutTimedWithFallback$TimeoutMainSubscriber$TimeoutTask.call(OnSubscribeTimeoutTimedWithFallback.java:191)
	at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
6

Any one who can help on this?
I have switched to use the synchronous API,I’m still blocked but at least now I’m getting TimeoutException

7

love it thank for the valuable information…

1 Like