We are using an older version of the Kafka connector. Recently we upgraded our Couchbase cluster from v6 to v7. After the upgrade we restarted the connector with stream value set to “SAVED_OFFSET_OR_NOW”. This is a new Couchbase cluster so no OFFSETS should exist right? We expected it to start from “NOW” but when I started the connector it went and pulled a lot more messages then we expected. Most likely from “Beginning”. Is this a bug in older versions of the connector? I am using v3.4.8. Does importing data from one Couchbase to another also carry over some sort of OFFSET information?
Hi Miguel! The offsets are saved in Kafka, so there were still offsets when the connector was started. However, I suspect the offsets were someone how invalid for the new cluster, so the connector “rolled back to zero” and streamed everything from the beginning.
If you want to resume from “NOW”, you can ignore the saved offsets by renaming the connector, or by deleting the offsets using the new REST API added in Kafka 3.6.0.