Kafka Connect - Update DCP Seed Nodes

Kafka Connector
1

Hi,
How can we update the number of Couchbase nodes of running connector instance?
We have our deployment in Kubernetes where the Couchbase server is containerized and while creating the connector instances we are using couchbase.seed.nodes: couchbase-service. This couchbase-service has multiple nodes and everything works fine. Depending on the load, during peak hours we need to add additional nodes to Couchbase cluster. How can we add these additional nodes automatically also to the same running connector instances without doing the restart of the Kafka Connect Pod?

We are using directly the Couchbase service (Kubernetes Service for Couchbase pods) and internally Couchbase client resolves this to couchbase.dcp.seed.nodes which has entries of previous nodes only. After we manually restart the Kafka Connect Pod, the connector resumes with updated Couchbase nodes. Shouldn’t Couchbase DCP Client update this config automatically whenever there is a change in number of Couchbase nodes in cluster? Are we missing some connector configuration to achieve this behaviour?

Thanks in advance.

2

Hi @patel-saket,

Are you saying the Couchbase Kafka connector is not seeing the new Couchbase nodes?

Thanks,
David

3

Hi @david.nault ,
Correct, the new nodes are not available to Kafka connect. They are visible only after a restart of Kafka Connect Pod is done.

Thanks.

4

Hi @patel-saket,

That’s strange, and not what I would expect – unless perhaps all the Couchbase Server nodes are going offline at the same time, and coming back with different addresses. Is that a possibility?

Do you have a Couchbase Enterprise Subscription License Agreement, by any chance? If so, it would be helpful to file a support ticket. We’ll investigate either way, but having a support ticket makes it easier to prioritize and allocate resources.

Thanks,
David

5

Thanks @david.nault I will create a support ticket.