Not able to read Couchbase(3.0.1) data through Kafka Connect(3.2.0)

mk12 · July 10, 2018, 11:06am

Hi,
We are using below configurations of database and connect

Couchbase 3.0.1 (42 machine cluster)
Couchbase Kafka connect (3.2.0 and 3.0.0)

We were able to read records through couchbase kafka connect for some time now.

When we restarted our connect cluster to read all the data again, we are not getting any db records pushed through kafka connect. We are seeing just below logs at connect side and no errors.

[2018-07-10 16:23:15,115] INFO WorkerSourceTask{id=cb-connector1-4} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
[2018-07-10 16:23:15,115] INFO WorkerSourceTask{id=cb-connector1-4} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:345)

FYI
We have XDCR enabled at Couchbase cluster.
We tried both in Standalone and distributed mode.

When we are starting kafka connector cluster, we could see in Couchbase UI that new DCP connections are being formed but no data flowing. Let me know if any other info needed.

david.nault · July 10, 2018, 6:37pm

Hi @mk12,

The Kafka Connect framework periodically saves a “bookmark” of which Couchbase changes have been published to Kafka. Is it possible you’re not seeing any new Kafka messages because there haven’t been any changes in Couchbase since the last time the connector saved a bookmark?

You could verify this in standalone mode by deleting the offset storage file. Its location is specified by the worker (not connector) configuration property offset.storage.file.filename and defaults to /tmp/connect.offsets . (In distributed mode, the offsets are stored in a Kafka topic. We could dig up the details if necessary.)

Version 3.2.2 of the connector introduced a configuration property couchbase.stream_from which you can set to BEGINNING if you want to ignore any saved state. Remembering our earlier conversation, I believe connector 3.2.2 is the most recent version that’s still compatible with Couchbase 3.0.1.

Thanks,
David

mk12 · July 11, 2018, 3:31pm

Hey @david.nault,
Version 3.2.2 with config couchbase.stream_from worked for us.
Thanks for quick reply.

vs07 · September 3, 2018, 4:33pm

Hi @david.nault,

kafka connect stopped streaming with this in logs :
[2018-09-03 12:51:23,943] INFO Starting to Stream for 35 partitions (com.couchbase.client.dcp.Client:115)
[2018-09-03 12:51:23,946] WARN Received rollback for vbucket 342 to seqno 185925 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,947] WARN Received rollback for vbucket 343 to seqno 185814 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,947] WARN Received rollback for vbucket 344 to seqno 185224 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,948] WARN Received rollback for vbucket 345 to seqno 184133 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,948] WARN Received rollback for vbucket 346 to seqno 181795 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,948] WARN Received rollback for vbucket 347 to seqno 186086 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,949] WARN Received rollback for vbucket 348 to seqno 185535 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,949] WARN Received rollback for vbucket 349 to seqno 185968 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,950] WARN Received rollback for vbucket 350 to seqno 184557 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,950] WARN Received rollback for vbucket 351 to seqno 181226 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,951] WARN Received rollback for vbucket 352 to seqno 186283 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,951] WARN Received rollback for vbucket 353 to seqno 185233 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,951] WARN Received rollback for vbucket 354 to seqno 183598 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,952] WARN Received rollback for vbucket 355 to seqno 185833 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,952] WARN Received rollback for vbucket 356 to seqno 185027 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,953] WARN Received rollback for vbucket 357 to seqno 185709 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,953] WARN Received rollback for vbucket 358 to seqno 184611 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,953] WARN Received rollback for vbucket 359 to seqno 214112 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,954] WARN Received rollback for vbucket 360 to seqno 213769 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,954] WARN Received rollback for vbucket 361 to seqno 212166 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,955] WARN Received rollback for vbucket 362 to seqno 181507 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,955] WARN Received rollback for vbucket 363 to seqno 182636 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,955] WARN Received rollback for vbucket 364 to seqno 181643 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,956] WARN Received rollback for vbucket 365 to seqno 185530 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,956] WARN Received rollback for vbucket 366 to seqno 184469 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,956] WARN Received rollback for vbucket 367 to seqno 182833 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,957] WARN Received rollback for vbucket 368 to seqno 184627 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,957] WARN Received rollback for vbucket 369 to seqno 182510 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,958] WARN Received rollback for vbucket 370 to seqno 185868 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,958] WARN Received rollback for vbucket 393 to seqno 1300 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,959] WARN Received rollback for vbucket 394 to seqno 542 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,959] WARN Received rollback for vbucket 395 to seqno 8869 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,959] WARN Received rollback for vbucket 396 to seqno 0 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,960] WARN Received rollback for vbucket 397 to seqno 0 (com.couchbase.client.dcp.Client:160)
[2018-09-03 12:51:23,960] WARN Received rollback for vbucket 398 to seqno 0 (com.couchbase.client.dcp.Client:160)

So this happened when we tried to increase connector tasks. The procedure I followed was:

Pause the connector and ensure all tasks are in paused state
Updated connector configuration with increased tasks.max
Resume the connector

I went through the thread and issue regarding Connector stops streaming after rollback. We don’t mind if we stream the events again (duplicate in kafka). Looks like you have fixed this in recent release. But unfortunately we can’t use later versions as our couchbase cluster is old(we discussed this before).

I am hoping you can shed some light on scenarios where rollback will happen.

david.nault · September 17, 2018, 4:38pm

Hi @vs07,

Here’s a blog post that talks about rollbacks and lists some possible causes:

https://blog.couchbase.com/couchbase-dcp-rollback-qa-tests/

Thanks,
David