[JCBC-208] HTTP flush needs to be async

Wed, 9 Jan 2013 05:13:02 -0600

Since flush can take a long time, it should be possible to handle this async.

[JCBC-28] refactor the entire cluster stream connection

Thu, 31 Jan 2013 09:57:54 -0600

Because of the codebase's legacy, the handling of the Bucket and Configuration is rather odd. It used to exist outside the client to serve a different purpose. At that time, not changing the client internals was desirable.

Fast forwarding to now, the internals should be updated to have the NodeLocator or the connection abstract away much of the configuration details.

[JCBC-209] Clarify the sync/async nature of operations in the docs

Fri, 11 Jan 2013 00:43:28 -0600

Clarify the sync/async nature of operations in the docs

[JCBC-33] Reconfiguration strategy used in TapConnectionProvider can lead to temporary deadlock of the ConfigurationProvider thread

Tue, 13 Nov 2012 07:47:53 -0600

Let's assume for now that a new node has been added to the cluster.

1. When a configuration change is detected, the reconfigure() method of TapConnectionProvider is called.
2. TapConnectionProvider will call ((CouchbaseConnection)conn).reconfigure(bucket);
3. Inside CouchbaseConnection reconfigure() the new server will be found and added to the list newServers
4. Then createConnections(newServers) will be called in the parent class MemcachedConnection
5. Depending on the log level you'll see the message logged from getLogger().info("Added %s to connect queue", qa);
6. Then the code will hang at the line: https://github.com/couchbase/spymemcached/blob/master/src/main/java/net/spy/memcached/MemcachedConnection.java#L155

qa.setSk(ch.register(selector, ops, qa));

The code will hang until another packet is received on the channel. This is the expected behavior in Java NIO. The recommended practice is perform registrations from the same thread as selects. In this case we're registering from the thread that was monitoring for configuration changes, and selecting from the main run loop of the MemcachedConnection.

See http://stackoverflow.com/questions/1057224/thread-is-stuck-while-registering-channel-with-selector-in-java-nio-server

It's not a huge problem for us, because even in an idle situation we eventually receive a NOOP. However, this combined with another bug I was hitting and made it really hard to troubleshoot. I'd suggest we look at ways to avoid this problem.

It's also possible this is a bug in Spy and not the Java client, but I haven't studied how createConnections() is used in other contexts within spy, so it might just be how we use it when reconfiguring from the java client.