Bucket exploration issues after upgrading to Couchbase 7.1.0

Hello!

A couple of days ago I have upgraded from community-7.0.2 to community-7.1.0 and from enterprise-7.0.3 to enterprise-7.1.0 for both of these I have noticed the following problems:
1, The Documents tab does not work anymore even though there are multiple buckets each filled with scopes and collections. Newly created buckets, scopes and collections also do not appear here if there is already a pre-existing bucket on the cluster, but if I delete all of the buckets and then create a new one the Documents tabs works again.


2, Although queries do work the “Explore Your Data” part is empty.

3, I have also noticed that there were random duplicate indexes (With identical name and definition), although I could not validate if those were created after the upgraded.

Is there a way to solve this without deleting the buckets? Currently only the development Clusters were upgraded and deleting the buckets there is fine, but eventually we want to upgrade the production cluster as well where that is not the case.

The cluster are runing on kubernetes using the official Couchbase dockers and the upgrade was done by changing the image tag in the Helm Chart.

Hi @horvath-martin can you explain the upgrade process here from 7.0.3 → 7.1.0 enterprise? The best way for us to understand what’s happening would be for you to share your helm-overrides along output from the resource collection tool. And you mentioned having a pre-existing bucket, was this created and managed by operator?
Also note that operator does not manage indexes and so it’s possible that duplicate indexes is outside of the scope of operator.

@horvath-martin Have you tried clearing the browser cache? It is possible there are pieces of old UI code still cached because their paths are the same as those of new UI code.

@Kevin.Cherkauer I have check it out in multiple browsers with cache cleared and it does not change anything. I suspect the old buckets are the cause. On one of our clusters I deleted all the buckets and created a new one and everything worked fine after that.

@tommie

  • The buckets were created by hand on version 7.0.3 in case of enterprise and on version 7.0.2 in case of community.
  • The scopes, collections and indexes are created by our application dynamically. (Through the Java and Scala SDK)
  • We use our own helm chart and I simply changed the image tag.
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: 
  namespace: 
spec:
  chart:
    spec:
      chart: couchbase
      sourceRef:
        kind: HelmRepository
        name: 
        namespace: 
      version: '0.5.5'
  interval: 30m0s
  rollback:
    recreate: true
    force: true
  values:
    image:
      repository: "couchbase"
      tag: community-7.1.0
      pullPolicy: Always
    persistence:
      size: 50Gi
      class: local
    replicas: 4
    resources:
      requests:
        memory: 27500Mi
        cpu: 1
      limits:
        memory: 27500Mi
        cpu: 7
    vpn:
      enabled: false

@horvath-martin When logged into the UI, what roles does the user have? There may be some permissions issues in retrieving the metadata to list the buckets/scopes/collections. The permissions-checking code changed between 7.0.3 and 7.1.

Do you see any errors in the browser’s error console?

@eben The account has Full Admin role and the following error message appears when opening the “Documents” or the “Query” menu.

Error: Uncaught (in promise): TypeError: this.rbac.cluster.bucket[bucketName] is undefined

@horvath-martin Thank you for the error message - I can see the place in the code where that is coming from.

I’m going to try to reproduce the problem, can you tell me more details about the upgrade:

  • Have you upgraded all nodes of the cluster to 7.1.0, or are some nodes still on 7.0.3?
  • It sounds like you are using multiple collections. Are you using the “_default” collection at all?

Can you try running the following curl command against the upgraded system:

curl -X POST -u <login>:<password> http://<server>:8091/pools/default/checkPermissions --data 'cluster.bucket[<bucket name>].data.docs!upsert'

@eben

  • All of the nodes are upgraded to 7.1.0.
  • Each bucket has a _default scope and in it a _default collection this is not used by our application, unless the Java SDK use it under the hood. (eg.: transactions, indexing or queries)
  • Our application creates its own scopes and collections, these scopes do not contain a _default collection.
  • Through the web UI, I can see that in some of the buckets the _default collection is not empty containing between 100-1000 items.

The curl commands returns with:

{
  "cluster.bucket[<bucket name>].data.docs!upsert": true
}

Not sure why, but not matter what I use as the bucket name true is returned.

@horvath-martin I’ve been trying to reproduce the issue without success.

Can you tell me how many nodes your cluster has? Which OS are you running on?

@eben Our cluster has 4 nodes and running on Kubernetes. Kubernetes is running on the following OS:

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.2
VERSION_ID=3033.2.2
BUILD_ID=2022-02-03-2047
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.2 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org/"
FLATCAR_BOARD="amd64-usr"

We have also uploaded Logs & Diagnostic Information to the forum under my username. Notify me if you cannot find it.

@eben I have managed to find and debug the code where the error happens.

this.bucketList.forEach(bucketName => {
    if (this.rbac.cluster.bucket[bucketName].data.docs.read ||
        this.rbac.cluster.bucket[bucketName].data.docs.upsert ||
        this.rbac.cluster.bucket[bucketName].data.docs.delete ||
        this.rbac.cluster.collection[bucketName + ':.:.'].data.docs.read ||
        this.rbac.cluster.collection[bucketName + ':.:.'].data.docs.upsert ||
        this.rbac.cluster.collection[bucketName + ':.:.'].data.docs.delete ||
        this.rbac.cluster.collection[bucketName + ':.:.'].n1ql.udf.manage ||
        this.rbac.cluster.collection[bucketName + ':.:.'].n1ql.udf_external.manage)
        this.permittedBucketList.push(bucketName);
});

Example data:

bucketName: "x.y.z"
this.rbac.cluster.bucket["x.y.z"]: undefined
this.rbac.cluster.bucket["x"]["y"]["z"]: Object

Our buckets’s name contain the ‘.’ character multiple times and it seems like the “this.rbac.cluster.bucket” payload contains a nested map with the keys being the bucket name splitted on the ‘.’ character.

I have also checked the “this.rbac.cluster.collection” and it contains the correct keys eg.: “x.y.z:.:.”.

Are ‘.’ in the bucket name not supported anymore or is this a bug?

Thank you @horvath-martin! I tried reproducing the problem, but did not know that your bucket names included ‘.’ characters. Since ‘.’ characters are allowed, this should be supported. I’m investigating how to get that working.

Hello again @horvath-martin,

I found a fix for the bug, and was able to get the fix in to release 7.1.2, which should be coming out some time soon. Thank you for all the debugging!

-Eben