Any resources for how to update Elasticsearch mapping when using the CB connector on Kubernetes?

Elasticsearch Connector
1

I wish to change field types in my Elasticsearch mapping. According to their recommendation, I would need to delete, then rebuild the index to make this work. I do this on the ES side, which went fine, but the connector did not re-replicate the documents over. I assume this has to do with the checkpoint information still telling the connector that everything is up to date?

I would recommend adding this information here https://docs.couchbase.com/elasticsearch-connector/current/autonomous-operations.html. I don’t think that most people get their mappings right the first time. Maybe it’s already documented somewhere, but I just haven’t been able to find it.

Any guidance will be appreciated.

2

Hi @sithembisophp ,

I assume this has to do with the checkpoint information still telling the connector that everything is up to date?

Yes, you’re absolutely correct. We’re working on a FAQ document to address common questions, and I’ll make sure this gets added to the list. In the mean time…

If you’re running the connector in Autonomous Operations Mode with Consul, this command will cause the connector to re-replicate all the documents:

cbes-consul checkpoint-clear --group=<group-name>

Reference: Autonomous Operations Mode – Connector Management Commands

You can safely run the cbes-consul checkpoint-clear command at any time, even while the connector is running.

If you’re not using Consul, and are instead running the connector in standalone mode with the cbes command, the command to clear the checkpoints is:

cbes-checkpoint-clear

Reference: Resetting the Connector

For the cbes-checkpoint-clear command, make sure the connector workers are all stopped before clearing the checkpoints.

Here’s what the docs say about which commands require stopping the connector first and which don’t:

All of the cbes-consul checkpoint-* commands may be performed at any time, even when workers are running. Just be careful not confuse them with the cbes-checkpoint-* commands, which should only be used when all workers in the group are stopped.

Reference: Tips & Tricks

Thanks,
David

3

Thank you for your response @david.nault .
I’m running the connector on Kubernetes as a StatefulSet. I’m not using Consul. So, I don’t have a way to stop the Pods and run cbes-checkpoint-clear. Do you reckon the best way to do this is to just redeploy it? Or is it safe to run it while the Pods are up?

4

Hi @sithembisophp ,

Probably not safe to do while the pods are up. The safe thing is probably to undeploy the pods (assuming this kills the processes inside; I don’t know much about k8s), then clear the checkpoints, and finally redeploy the pods.

It’s important for the connector processes to be completely restarted, not just paused.

Thanks,
David

5

Noted. This will stop the processes completely. Removing the volumes also will result in a clean installation. I will post the feedback here if I come across anything out of the ordinary. Thank you for your help.

1 Like