Unable to get deadletterqueue topics to work in CB Kafka source connector

Kafka Connector
1

I’d like to be able to use the deadletterQueue part of kafka connect with the couchbase connectors. This is my first POC using these connectors.

I’ve added the following to my connect-couchbase-source.properties:

tasks.max=1

# * Configure deadletter topic for error handling
errors.deadletterqueue.context.headers.enable = true
errors.deadletterqueue.topic.name = deadLetters_couchbase_source
errors.deadletterqueue.topic.replication.factor = 1
errors.log.enable = true
errors.log.include.messages = false
errors.retry.delay.max.ms = 60000
errors.retry.timeout = 0
errors.tolerance = all

But the topic does not get created and I can find no error messages.
Have I missed something or do the couchbase connectors not support the deadletterQueue topic on errors?

Other than that the connector seems to be functioning, but I’ve yet to try using the SMT feature of connect.

2

Hi Larry,

The Kafka Connect dead letter queue is only for sink connectors. Are you worried about a specific issue with the Couchbase source connector?

Thanks,
David

3

So can I do SMT operations in the source connector? and if they encounter errors I can’t push those records to an alternate topic for analysis or a possible alert?

4

Hi Larry,

So can I do SMT operations in the source connector? and if they encounter errors I can’t push those records to an alternate topic for analysis or a possible alert?

Yes, SMTs can be used with source connectors. I have to admit I’m not up to speed on best practices for SMT error handling. I bet someone over here might have an idea: Kafka Connect - Confluent Community

Or, if you prefer, you can write a custom SMT that does the transformation in code. This gives you complete control over error handling, changing the destination topic, etc.

Thanks,
David