Multiple buckets data in different topics of kafka


is it possible to have multiple buckets pushing data in same kafka cluster but on different topics.

If yes what configurations should be used for the same.

Pankaj Sharma

Hi Pankaj,

The current version of the connector (3.2.0) is limited to reading from a single bucket.

However, it is possible to run multiple instances of the connector, each configured with a different bucket/topic.


Can you provide some link or configuration help in it?

I’d recommend following the The Kafka Connect Couchbase Quickstart guide, which shows how to configure and run the connector. Once you’ve got the first connector instance running, make a copy of the file and edit the, connection.bucket and name properties (name is arbitrary, and must uniquely identify the connector instance). Then follow the steps to run the connector again, but this time use the modified config. Unless you stopped the first connector instance, you should now have two connector instances running, each reading from a different bucket.


Hi @david.nault
When i tried to run second instance of Kafka connector, then it gave me error .
FAILED org.eclipse.jetty.server.Server@565e7091: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212) Address already in use
at Method)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(
at org.eclipse.jetty.server.ServerConnector.doStart(
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(
at org.eclipse.jetty.server.Server.doStart(
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(
at org.apache.kafka.connect.runtime.Connect.start(
at org.apache.kafka.connect.cli.ConnectStandalone.main(

Hi @manya.goyal, source connector or sink connector?

It’s a source connector

Hi @manya.goyal, @pankaj.sharma,

I overlooked an important detail documented in Running Workers - Standalone Mode:

If you run multiple standalone instances on the same host, there are a couple of settings that must be unique between each instance:

  • - storage for connector offsets, which are stored on the local filesystem in standalone mode; using the same file will lead to offset data being deleted or overwritten with different values
  • rest.port - the port the REST interface listens on for HTTP requests

The “Address already in use” exception occurs because the two connector workers are configured with the same value for rest.port.


Hi @david.nault
Thank you so much for the detailed info! I already figured out the solution for “address already in use” Problem by changing rest.port
Though i will also include property in my config.
Thanks once again.

1 Like

For posterity, another way to run multiple instances of the Couchbase connector is to specify multiple *.properties files when starting the worker process, like: \