Helm Chart Deletion of Nodes not working as expected

I am currently experimenting with different cluster architectures, when I first set up a production cluster I had 3 combined nodes that had data + query + index + eventing services in place for all of them. The original operator configuration is like this:

servers:
default:
size: 3
services:
- data
- index
- query
- eventing
volumeMounts:
default: couchbase-volume

When this configuration hit performance problems I added in separate data, query and index nodes to follow best practices. This seems to have helped with the performance issues, but now I can’t remove the original default node. The new configuration:

servers:
default:
size: 1
services:
- data
- index
- query
- eventing
volumeMounts:
default: couchbase-volume
datanodes:
size: 2
services:
- data
volumeMounts:
default: couchbase-volume
querynodes:
size: 2
services:
- query
volumeMounts:
default: couchbase-volume
indexnodes:
size: 2
services:
- index
volumeMounts:
default: couchbase-volume

I want to remove the original default node, which should be possible according to this documentation. Syntaxes that I’ve tried:

  1. Remove the “default” section completely
  2. Scale the “default” section to 0 nodes
  3. Alter the “default” section to have the name “eventingnodes” and change this section to only have the eventing service

None of these syntaxes have worked for me, when I try to apply the chart changes I see the error:

cannot patch “oaf-couchbase” with kind CouchbaseCluster: admission webhook “oaf-couchbase-couchbase-admission-controller.couchbase.svc” denied the request: validation failure list:
spec.servers[1].services in body cannot be updated

This is with couchbase version 6.6.0 and the helm chart operator version 2.1.0. Any suggestions for how to remove this node without destroying/recreating the cluster?

The server class name is a key used to keep track of modifications to groups of servers. For some things, like the services, these cannot be modified by Couchbase server at runtime, so that’s what that error is trying to tell you.

The correct way to pull this off is, add the new server classes with different names to default, and remove the old default class, all at the same time. If you do so what error does it give you? If it does, that is.

I tried to switch from the current working configuration to this one (replacing “default” with “eventingnodes”):

servers:
    eventingnodes:
      name: eventingnodes
      size: 1
      services:
        - eventing
      volumeMounts:
        default: couchbase-volume
    datanodes:
      name: datanodes
      size: 3
      services:
        - data
      volumeMounts:
        default: couchbase-volume
    querynodes:
      name: querynodes
      size: 2
      services:
        - query
      volumeMounts:
        default: couchbase-volume
    indexnodes:
      name: indexnodes
      size: 2
      services:
     - index
   volumeMounts:
     default: couchbase-volume

but I encountered the same error as above:

spec.servers[1].services in body cannot be updated

Is that the syntax you mean?

Yeah looks good. Now it’s suggesting there is a problem with your datanodes class changing. Can you post the output from:

kubectl get cbc -o yaml

so I can compare with what’s already on the system? Just the spec.servers bit will suffice.

Sure, thanks for your help with this. Here’s the output from “get cbc”:

servers:  
- name: datanodes
  pod:
    metadata:
      creationTimestamp: null
    spec:
      containers: []
  resources:
    limits:
      cpu: "4"
      memory: 14Gi
    requests:
      cpu: "4"
      memory: 14Gi
  services:
  - data
  size: 2
  volumeMounts:
    default: couchbase-volume
- name: default
  pod:
    metadata:
      creationTimestamp: null
    spec:
      containers: []
  resources:
    limits:
      cpu: "4"
      memory: 18Gi
    requests:
      cpu: "4"
      memory: 18Gi
  services:
  - data
  - index
  - query
  - eventing
  size: 1
  volumeMounts:
    default: couchbase-volume
- name: indexnodes
  pod:
    metadata:
      creationTimestamp: null
    spec:
      containers: []
  resources:
    limits:
      cpu: "2"
      memory: 5Gi
    requests:
      cpu: "2"
      memory: 5Gi
  services:
  - index
  size: 2
  volumeMounts:
    default: couchbase-volume
- name: querynodes
  pod:
    metadata:
      creationTimestamp: null
    spec:
      containers: []
  resources:
    limits:
      cpu: 600m
      memory: 1Gi
    requests:
      cpu: 600m
      memory: 1Gi
  services:
  - query
  size: 2
  volumeMounts:
    default: couchbase-volume

I notice that the cbc output is in the format:
-datanodes
-default
-indexnodes
-querynodes

So I attempted to change the order of my helm chart values to match:
-datanodes
-eventingnodes (formerly default)
-indexnodes
-querynodes

I still see the same error (regardless of whether datanodes is first in the values.yml file or eventingnodes is first):

spec.servers[1].services in body cannot be updated

@tommie can you have a look at this? Problem is the “default” is apparently being updated when it’s removed from the helm configuration. I’ve checked the DAC code, this should only fire when a new name aliases with an existing one.

If default is being removed, how can it possibly match against the existing one! Dunno if helm is doing something weird here and not deleting it properly. Do you know of any magical Helm runes to debug this further?

Yea it does feel counter-intuitive, but helm is override-only so the way to remove things is to override with ‘empty’ values. Try:

  servers:
    default: null
    datanodes: ...

Also, you can omit the name key if the name of the class is same as the top-level key.

1 Like

Thanks, that syntax worked.