Today we ran into an issue where all our clients ( Java SDK ) began getting Authentication errors:
com.couchbase.client.core.endpoint.Endpoint - [localhost][KeyValueEndpoint]: Authentication Failure: Authentication Failure
This seemed to stem from changing the cluster admin password the day prior. The order of events:
- Monday - Reset cluster admin password using “couchbase-cli reset-admin-password”. ( Did not change password for the application account and access was still working)
- Tuesday - Removed one node from the cluster for monthly patching and upon removal / rebalance, the entire cluster began giving “Authentication Failure” errors for our application account.
- Immediately logged into UI and reset the application password to the exact same password we were using before, and everything was able to connect again.
This leads me to believe that somehow the reset-admin-password also reset the local application account we created, but something must have been stored in memory until a rebalance was issued.
Couchbase Version: Community Edition 5.1.1 build 5723
OS Version: CentOS 7.6
Hopefully someone else has seen this issue and can advise how to proceed. Up until this point, we have had very good success with Couchbase.