How to separate backfill and live change events with Couchbase Kafka Connector?

Kafka Connector
,
1

Hi Team,
I’m using the Couchbase Kafka Source Connector to ingest data from a bucket into Kafka.

  • I started with "couchbase.stream.from": "BEGINNING", which successfully ingested the full backfill of documents.

  • After that, I keep seeing a large number of records on the Kafka topic, but I can’t clearly tell which ones are new live changes versus replay/backfill.

My goal is:

  1. Ingest all existing documents once (backfill).

  2. Then continue streaming only new changes (inserts/updates/deletes) going forward.

My questions are:

  1. What is the correct/best practice approach to separate backfill and live streaming?

  2. How can I reliably detect that the backfill phase is complete, so I know when to stop the first connector? (e.g., using cbstats dcp or other Couchbase metrics?

Any examples, or recommended configurations would be appreciated!

2

use SAVED_OFFSET_OR_BEGINNING.

The first time.the connector runs, it will start from the beginning and save the offsets as they are streamed. After that, it will start streaming from where it left off. Be sure to use the latest version of the connector as it now handles a condition that could result in a backlog of dcp events.