After doing this, when we created a new Couchbase instance with 4 pods, only one pod was getting created.
We opened a ticket with RedHat to diagnose this issue further as we were not seeing any errors.
While working with RedHat, we notice that the Couchbase Operator was installed on the openshift-operator project instead of the inf-auto project where we created the Couchbase cluster instance. I remember selecting inf-auto when I installed it the first time, so this was unexpected.
We removed the operator and re-installed it in the inf-auto project.
When we tried to create a new Couchbase cluster instance, no pods are getting created and we see the following error:
{“level”:“info”,“ts”:1619812059.4318697,“logger”:“cluster”,“msg”:“Cluster does not exist so the operator is attempting to create it”,“cluster”:“a-couchbase-test/cb-example-test4”}
Our openshfit admin installed our operator since we do not have the permission. This is the summary from them, however we do not have a working couchbase cluster. We really do not understand why the network policy change could affect couchbase cluster.
That’s the hint we need… let me explain. In the distant past, you needed to fill in the fsGroup correctly or persistent volumes wouldn’t work. After every user didn’t fill this in, we decided to try do it for you with the dynamic admission controller. On OCP this interrogates the namespace that the cluster lives in and extracts the fsGroup from the annotations, which makes me suspect that the dynamic admission controller isn’t working correctly. You can manually set the fsGroup using these instructions Persistent Volumes | Couchbase Docs
I manually reset the fsGroup, all the pods come up. Thanks for your help.
I sent the following to our openshift admin, questioned the DAC is not there.
Check the Status of the Operator
You can use the following command to check on the status of the deployments:
$ oc get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
couchbase-operator 1/1 1 1 8s
couchbase-operator-admission 1/1 1 1 8s
CONSOLE
The Operator is ready to deploy CouchbaseCluster resources when both the DAC and Operator deployments are fully ready and available
root@usapprshilt100:/Automation/projects/openshift #oc project a-couchbase-test
Now using project “a-couchbase-test” on server “https://api.ivz-ocp-poc.ops.invesco.net:6443”.
root@usapprshilt100:/Automation/projects/openshift #
root@usapprshilt100:/Automation/projects/openshift #oc get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
couchbase-operator 1/1 1 1 3d3h
root@usapprshilt100:/Automation/projects/openshift #n the poc,
I did not see the couchbase DAC running, couchbase-operator-admission is missing
But our admin mentioned if in our dev env, when they installed couchbase for all named space, they did not see DAC running, but everything is working fine. Now they want to install the couchbase operator only for our name space. this is where the problem pods are not coming up. So when the operator is installed for all the name space, you do not need DAC ?
No the DAC needs to always be installed. We recommend it’s run in the default cluster mode, and therefore you only need one installed, in any namespace.
When we install it from GUI interface, according to our openshift admin, after he click install, the DAC is not installed. I could try to install it using the yaml file according to the instruction on the operator documents, however, I think that my permission as the admin of the name space is not good enough to finish the installation, it will still need openshift cluster admin role to install it ?
Hello guys, i feel i had same issue like @lukq 's issue.
Because i’ve installed the operator, only one instance(pod) from the cluster start running & it keep restarting. Checking events in desc order, this is what we had:
94s Warning MemberCreationFailed couchbasecluster/couchbase-cluster New member couchbase-cluster-0000 creation failed
89s Normal ServiceCreated couchbasecluster/couchbase-cluster Service for admin console `couchbase-cluster-ui` was created
Also i noticed that there is no deployment nor statefulset. Seems like the operator directly managing pods.
May be because we are running very recent version of the operator: