Details
Description
We have a small round robin pool of MemcachedClient objects.
On application shutdown we do call shutdown on each MemcachedClient,but it hangs and application shutdown hangs too.
Do you have any ideas why?
I have a thread dump
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700b24368> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at net.spy.memcached.MemcachedClient.waitForQueues(MemcachedClient.java:2120)
at net.spy.memcached.MemcachedClient.shutdown(MemcachedClient.java:2073)
On application shutdown we do call shutdown on each MemcachedClient,but it hangs and application shutdown hangs too.
Do you have any ideas why?
I have a thread dump
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700b24368> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at net.spy.memcached.MemcachedClient.waitForQueues(MemcachedClient.java:2120)
at net.spy.memcached.MemcachedClient.shutdown(MemcachedClient.java:2073)
But application doesn't use async operations, all calls are sync and app waits for result, to make sure data was persisted.
When i do call simple client.shutdown(30, TimeUnit.SECONDS) it hangs(see bug description).
When i added timeout for shutdown call, like:
future = executor.submit(new Callable<Boolean>() {
public Boolean call() throws Exception {
return client.shutdown(30, TimeUnit.SECONDS);
}
});
future.get(1, TimeUnit.MINUTES);
Application able to exit with messages :
WARN 22 Feb 2012 12:35:21,337 [Memcached IO over {MemcachedConnection to xxxx} - SHUTTING DOWN (waiting)] (?:?) - Closing, and reopening {QA sa=xxx, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 10 Opaque: 224540867, topWop=null, toWrite=0, interested=1}, attempt 0.
WARN 22 Feb 2012 12:35:21,339 [Memcached IO over {MemcachedConnection to xxxxx } - SHUTTING DOWN (waiting)] (?:?) - Discarding partially completed op: Cmd: 10 Opaque: 224540867
Cmd:10 it's a NoopOperationImpl
Does it mean that Noop hangs?