Couchbase Kafka connector

Kafka Connector
1

Hi,

We use the Kafka connector (https://docs.couchbase.com/kafka-connector/3.4/index.html) in the sink mode.

We have a situation that requires to create 2 Couchbase documents as output (instead of 1, as normal), from 1 Kafka message as input.

Is this somehow possible to achieve ?

Best regards,
Vlad

2

Hi Vlad,

Couchbase Kafka Connector 3.4 only supports creating a single document from a Kafka message. If you want two documents, you’d have to split the Kafka message upstream (maybe using Kafka Streams? I don’t know the details of how that would work, just that it should be possible).

I’ve filed enhancement request KAFKAC-153 for this feature, which you can watch on Couchbase Jira. If you happen to be an Enterprise Edition customer, please also reach out to your support contact so they can start an escalation referencing this issue.

Cheers,
David

3

Thank you very much David for creating the enhancement request.
I should add, to make our case clearer, that we need to get 2 different Couchbase documents, obtained by applying 2 different custom transformations.
Do you think this can be achieved in the Kafka Connector or it has to be done in the Kafka Connect itself ?
Anyway, we are looking forward to see it implemented.

Best regards,
Vlad