Sent duplicate event to Kafka via Kafka Connect

Kafka Connector
1

Hi,

We encountered an interesting problem recently. The kafka connector sent duplicate events to kafka until we restarted the connector app. We did not see any problem in db or kafka at the time of the problem.

We are using Couchbase Kafka Connect 4.0.2 version with distributed mode to capture events and send them to Kafka. And also XDCR enabled at Couchbase cluster. We are using Couchbase Enterprise Edition 6.6 version.

Kafka connect configuration as below:

{
“connector.class”: “com.couchbase.connect.kafka.CouchbaseSourceConnector”,
“couchbase.persistence.polling.interval”: “0”,
“tasks.max”: “3”,
“couchbase.compression”: “ENABLED”,
“connection.timeout.ms”: “2000”,
“transforms”: “ignoreDeletes,deserializeJson”,
“couchbase.source.handler”: “com.couchbase.connect.kafka.handler.source.RawJsonSourceHandler”,
“couchbase.seed.nodes”: “nodes”,
“couchbase.bucket”: “bucket”,
“couchbase.username”: “user”,
“couchbase.stream.from”: “SAVED_OFFSET_OR_NOW”,
“value.converter.schemas.enable”: “false”,
“event.filter.class”: “com.couchbase.connect.kafka.filter.AllPassFilter”,
“name”: “name”,
“couchbase.password”: “pass”,
“value.converter”: “org.apache.kafka.connect.json.JsonConverter”,
“transforms.deserializeJson.type”: “com.couchbase.connect.kafka.transform.DeserializeJson”,
“couchbase.topic”: “topic”,
“transforms.ignoreDeletes.type”: “com.couchbase.connect.kafka.transform.DropIfNullValue”
}

Is there any mistake in our config? What else should we check? Can anyone help?

Thanks,
Umit

2

Hey @umt_brbr,

I’ve routed your question to the SDK team. They will be able to have a look soon.

3

Hey @umt_brbr,

Would you be able to give a bit more context to the issue you’re having?
Do you have any database logs, or other log information you could share?
Did you have multiple instances of the Kafka connector running?

For example, as stated in the docs In some situations the source connector may “rewind” the document stream and restart from an earlier point in history. See Introduction | Couchbase Docs, and this can happen if there are underlying issues.

4

Hi @Nebasuke,

We have three instances of Kafka connector.

I will ask the logs from the db team but as far as I understand it’s possible that kafka connector works like this in some situations.

Do we know in which situations it works this way? Such as failover in Couchbase or what else?

Thanks