Filter documents in kafka connector bucket source

Kafka Connector
1

Hi,

I need to stream some of my bucket documents to a kafka topic. Does the kafka connector support this?
According to this: Source Configuration Options | Couchbase Docs I can only specify the bucket name, I assume that all documents in this bucket will be streamed to kafka?
How can I define which documents in the bucket get streamed to the kafka topic?

Regards,
Faris

2

@FarisAhmed welcome back.

There is no filter on the CB side, however you could use the filter function of Kafka Connect transformations. The filter function allows to include or exclude records that match the predicate. Please, see transforms/filter documentation and examples.

3

Hi @FarisAhmed,

The kafka-connect-couchbase GitHub repo has an examples directory with a custom-extensions project. One of the examples is for a custom filter. You could modify this to implement your filtering logic.

If you can upgrade to Couchbase 7 and Kafka connector 4.1, a cleaner solution might be to store the documents you want to stream in separate Couchbase collection(s), and specify the collection names in the couchbase.collections config property.

Thanks,
David