Restore from hibernation in another kubernetes cluster

Hello !

I use CAO 2.0.3 and create this minimal cluster

apiVersion: v1
kind: Secret
metadata:
  name: cb-example-auth
type: Opaque
data:
  username: QWRtaW5pc3RyYXRvcg== # Administrator
  password: cGFzc3dvcmQ=         # password
---
apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
  name: default
---
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: cb-example
spec:
  image: couchbase/server:7.0.3
  security:
    adminSecret: cb-example-auth
  buckets:
    managed: true
  volumeClaimTemplates: 
  - metadata:
      name: couchbase 
    spec:
      storageClassName: managed-premium 
      resources: 
        requests:
          storage: 1Gi
  servers:
  - size: 3
    name: all_services
    services:
    - data
    - index
    - query
    - search
    - eventing
    - analytics
    volumeMounts:
      default: couchbase

I wanted to move this couchbase cluster in another kubernetes cluster and I set it up in hibernate mode by setting spec.hibernate=true. Then I took a copy of the pvc and restore them in another cluster.

When I restart the cluster by setting spec.hibernate=false I get this error message in the logs of the operator

persistentvolumeclaims \"cb-example-0001-default-00\" already exists

a more complete trace

{"level":"error","ts":1656367663.5471344,"logger":"cluster","msg":"Pod addition to cluster failed","cluster":"default/cb-example","error":"fail to create member's pod (cb-example-0001): persistentvolumeclaims \"cb-example-0001-default-00\" already exists","stacktrace":"github.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).handleAddNode\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:746\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:307\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:260\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:172\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1656367663.5486817,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"fail to create member's pod (cb-example-0001): persistentvolumeclaims \"cb-example-0001-default-00\" already exists","stack":"github.com/couchbase/couchbase-operator/pkg/util/k8sutil.createPersistentVolumeClaim\n\tgithub.com/couchbase/couchbase-operator/pkg/util/k8sutil/pod_util.go:574\ngithub.com/couchbase/couchbase-operator/pkg/util/k8sutil.CreateCouchbasePod\n\tgithub.com/couchbase/couchbase-operator/pkg/util/k8sutil/pod_util.go:105\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).createPod\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/pod.go:37\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).createMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:168\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).addMember\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/member.go:236\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).handleAddNode\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:744\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*ReconcileMachine).exec\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/nodereconcile.go:307\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileMembers\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:260\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:172\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}

I probably missed something here, how the operator decide to recreate a new PVC or to reuse an existing one ?

Finally solved my issue, I forgot to recreate couchbase annotation on the pvc.