[SPY-79] Shutdown method hangs Created: 21/Feb/12  Updated: 29/May/13

Status: Open
Project: Spymemcached Java Client
Component/s: library
Affects Version/s: 2.7.3
Fix Version/s: .next
Security Level: Public

Type: Bug Priority: Critical
Reporter: Alexander Khimich Assignee: Michael Nitschinger
Resolution: Unresolved Votes: 0
Labels: hang, shutdown, spylib, spymemcached
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Membased 1.7.1
Spylib 2.7.3
Java 6

We have a small round robin pool of MemcachedClient objects.
On application shutdown we do call shutdown on each MemcachedClient,but it hangs and application shutdown hangs too.
Do you have any ideas why?

I have a thread dump

   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x0000000700b24368> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
        at net.spy.memcached.MemcachedClient.waitForQueues(MemcachedClient.java:2120)
        at net.spy.memcached.MemcachedClient.shutdown(MemcachedClient.java:2073)

Comment by Alexander Khimich [ 22/Feb/12 ]
You could probably think that it's waiting for Queues, for async operations.
But application doesn't use async operations, all calls are sync and app waits for result, to make sure data was persisted.

When i do call simple client.shutdown(30, TimeUnit.SECONDS) it hangs(see bug description).

When i added timeout for shutdown call, like:
future = executor.submit(new Callable<Boolean>() {
                    public Boolean call() throws Exception {
                        return client.shutdown(30, TimeUnit.SECONDS);
future.get(1, TimeUnit.MINUTES);

Application able to exit with messages :
WARN 22 Feb 2012 12:35:21,337 [Memcached IO over {MemcachedConnection to xxxx} - SHUTTING DOWN (waiting)] (?:?) - Closing, and reopening {QA sa=xxx, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 10 Opaque: 224540867, topWop=null, toWrite=0, interested=1}, attempt 0.
 WARN 22 Feb 2012 12:35:21,339 [Memcached IO over {MemcachedConnection to xxxxx } - SHUTTING DOWN (waiting)] (?:?) - Discarding partially completed op: Cmd: 10 Opaque: 224540867

Cmd:10 it's a NoopOperationImpl
Does it mean that Noop hangs?

Comment by Alexander Khimich [ 24/Feb/12 ]
Another observation it happens only with spymemcached Vbucket feature
But doesn't happens when you use node list like
Comment by Michael Nitschinger [ 29/May/13 ]
Assigning to .next for now until we have decided how to progress with it.
Generated at Wed Dec 17 17:27:44 CST 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.