Java client not aware about failed over node

We are using the latest version of couchbase-java-client (1.4.0-dp2).
Assume the following scenario:

  • We have a cluster with 2 nodes.
  • Client is able to read & write documents to a bucket.
  • create a couple of documents (one is active on node1, the other is active on node2).
  • the node1 goes down.
  • as expected, the client can retrieve only active documents from the node2 (which is up). The other documents are unavailable.
  • perform manual fail over on node1 from admin console.
  • after fail over is complete, the cluster is healthy and all documents are active on node2.

Expected behavior:
the client should be able to retrieve all the documents (including those which were active on failed node).

Actual behavior:
the client can retrieve only documents which were active on node2. The only way to retrieve all the documents is to create a new instance of the client.

We believe that the client is not informed about cluster topology change after manual fail over is complete (we see in logs that the client is retrying to connect to the failed over node). Is it a known issue?

After upgrading couchbase-java-client to 1.4.0, the problem still persist.

I have tried using the version 1.4.1 of the client and the problem seems to be fixed. Here is the relevant github discussion: