Java SDK - rebalance causes all operations to block
I've just rebalanced my eight-node cluster to take one node out, upgrade it to have four CPUs and then add it back in.
Upon adding it back in the seven machines (which all still have two CPUs) that were originally in the cluster are reporting timeouts.
Our healthcheck is failing to talk to Couchbase with the following error message (our healthcheck consists of getting a random GUID):
Timed out waiting to add Cmd: 0 Opaque: 28838565 Key: f000880b-9bc2-428a-879f-50044b07c6e5(max wait=10000ms)
All nodes are reporting the correct number of items. The usual number of gets and sets are being carried out against the cluster but from the Web-UI it appears that none are getting through.
Has anyone seen the Java SDK fail to pick up changes to the cluster following a rebalance?