Ingesting all documents from Couchbase with kafka connector?

Kafka Connector
,
1

Hi Team,
I am using Capella. And we have 300million documents in capella.
I’m using the Couchbase Kafka Source Connector to ingest data from a bucket into Kafka.

  • I wanted to copy all 300Million documents to kafka.But when I use kafka connect, I am seeing 10000Million messages in kafka distributed between 3 partitions. My requirement is to copy all documents to kafka, and live changes also to kafka.
  • I also want live changes to go to kafka.

how do I do this.

Any examples, or recommended configurations would be appreciated!

@mreiche - can you please help.

2

use SAVED_OFFSET_OR_BEGINNING.

The first time.the connector runs, it will start from the beginning and save the offsets as they are streamed. After that, it will start streaming from where it left off. Be sure to use the latest version of the connector as it now handles a condition that could result in a backlog of dcp events.

I wanted to copy all 300Million documents to kafka.But when I use kafka connect, I am seeing 10000Million messages in kafka distributed between 3 partitions.

There will be at least one message per existing document. Subsequent modifications will result in one or more subsequent messages. (multiple modifications may handled by a single message).

3

Thanks a lot. This approach worked for me

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.