Possible bug: indexing service frequently fails to create indexes for a programatically created bucket

[zoltan.zvara]

Start a Couchbase CE 7.0.2 or EE 7.0.3 (reproduced on both with the same frequency) on Kubernetes with 4GB RAM for data service and 2GB RAM for indexing service as follows.

The cluster is empty.

Using the Java API 3.2.5 and Scala API 1.2.5 & Scala 2.13.8, create a bucket with the following settings.

         scalaCluster
            .buckets
            .create(CreateBucketSettings(
              name = bucketConfiguration.bucketName,
              ramQuotaMB = bucketConfiguration.ramQuotaMB,
              flushEnabled = Some(bucketConfiguration.flushEnabled),
              numReplicas = Some(bucketConfiguration.numReplicas),
              replicaIndexes = Some(bucketConfiguration.replicaIndexes),
              bucketType = Some(bucketConfiguration.bucketType),
              ejectionMethod = Some(bucketConfiguration.ejectionMethod),
              maxTTL = None,
              compressionMode = None,
              conflictResolutionType = None,
              minimumDurabilityLevel = Some(bucketConfiguration.minimumDurabilityLevel)
            ))

After bucket creation, wait 30 seconds so that the bucket is properly initialized - just to make sure.

Programatically create scope oouOCeUc and collection sehcs-Whatever_e.

Then programmatically create the following index.

CREATE INDEX `PXPy_1646932173650-oouOCeUc-sehcs-Whatever_e-value` ON `PXPy_1646932173650`.`oouOCeUc`.`sehcs-Whatever_e`(`value`)

The index creation fails about 65 % of time either due to:

The server reported an issue with the underlying index {“completed”:true,“coreId”:“0x4c0c751600000002”,“errors”:[{“code”:12021,“message”:“Scope not found in CB datastore default:PXPy_1646932173650.oouOCeUc”,“retry”:false}],“httpStatus”:500,“idempotent”:false,“lastDispatchedFrom”:“10.240.0.10:50352”,“lastDispatchedTo”:“cbt-5-couchbase-0.cbt-5-couchbase.development.svc.sigma:8093”,“requestId”:155,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“9ad19128-1555-4f24-9b0d-37d68fdc3a99”,“statement”:“CREATE INDEX PXPy_1646932173650-oouOCeUc-sehcs-Whatever_e-value ON PXPy_1646932173650.oouOCeUc.sehcs-Whatever_e(value)”,“type”:“query”},“timeoutMs”:30000,“timings”:{“dispatchMicros”:39398,“totalDispatchMicros”:39398}}

or

Index already exists {“completed”:true,“coreId”:“0x4c0c751600000002”,“errors”:[{“code”:4300,“message”:“The index PXPy_1646932173650-oouOCeUc-sehcs-Whatever_e-value already exists.”,“retry”:false}],“httpStatus”:409,“idempotent”:false,“lastDispatchedFrom”:“10.240.0.10:50419”,“lastDispatchedTo”:“cbt-5-couchbase-0.cbt-5-couchbase.development.svc.sigma:8093”,“requestId”:900,“requestType”:“QueryRequest”,“retried”:0,“service”:{“operationId”:“fcebd508-9a22-43aa-b987-7bde8924ed82”,“statement”:“CREATE INDEX PXPy_1646932173650-oouOCeUc-sehcs-Whatever_e-value ON PXPy_1646932173650.oouOCeUc.sehcs-Whatever_e(value)”,“type”:“query”},“timeoutMs”:30000,“timings”:{“dispatchMicros”:50132,“totalDispatchMicros”:50132}}

(When you check in the UI, it does not exist.)

Or the index exists, but it reports an ERROR on the web UI.

However, after the bucket creation and waiting 30 seconds as previously, but running /pools/default/buckets/$bucket/controller/compactBucket programatically and then following up with scope, collection, and then index creation, indexes properly come up and created every time.

When I remove the compaction command after bucket creation, the program attempts to create the indexes no matter what. It continuously checks whether the index is created or online, waits and after a while runs the index creation again, for 5 minutes. When any time during the reattempts for the index creation, I go to the web UI and click on “Compact” for the said bucket, the index creation then properly finishes and the program then can verify that the index is created and online.

[Marco_Greco]

Yup, your complaint is absolutely valid.
The underlying issue is that currently DDL is asynchronous, so if you are creating the scope and collection via the SDK, or couchbase-cli, N1QL has to wait for the node coordinator to alert it that there is a new scope and collection.
We are working to make DDL synchronous.

The good news is that if you have a single query node, and you create scope and collection via N1QL (CREATE SCOPE oou0CeUc, etc), the N1QL node catches that, and you don’t need to wait.

[graham.pople]

Hi @zoltan.zvara
As a workaround for now, until the synchronous DDL is available, you can do what we do in our internal testing: poll the system:keyspace table until it contains the buckets, scopes or collections you need.
E.g. to wait for a particular keyspace (a bucket or collection) to be available for indexing:

    boolean ready = false;
    int guard = 100;

    while (!ready && guard != 0) {
      guard -= 1;
      String statement =
              "SELECT COUNT(*) > 0 as present FROM system:keyspaces where name = '" + keyspaceName + "';";

      QueryResult queryResult = cluster.query(statement);
      List<JsonObject> rows = queryResult.rowsAsObject();
      if (rows.size() == 1 && rows.get(0).getBoolean("present")) {
        ready = true;
      }

      if (!ready) {
        try {
          Thread.sleep(50);
        } catch (InterruptedException e) {
           throw new RuntimeException(e);
        }
      }
    }

    if (guard == 0) {
      throw new IllegalStateException("Query indexer is not aware of keyspaceName " + keyspaceName);
    }

You might need to make the SELECT more specific to check the bucket, scope and collection names all match yours exactly (this test code doesn’t need that specificity as we’re using UUIDs for collections).
You’ll probably want to catch and ignore the IndexExistsException too.

[zoltan.zvara]

@graham.pople for the bucket, collection, scope creation we wait for them to come online just as you outlined in your code snippet. Instead of COUNT(*) we sometimes get all results and find pending/online objects. Then we hit the indexing service with the index creation.

As we observed, waiting does not help in this case. Compation must be initiated manually or programmatically to get the bucket or indexing service working so that it can create indexes properly.

[Christophe_H]

We are working to make DDL synchronous.

hello,
Any idea when this bug will be fixed?

Thanks.

[mreiche]

The issue for index creation is Loading....