Our Kafka connector logged some WARNINGS that look like this:
"Received rollback for vbucket [vBucket_ID] to seqno 0 ; requested start offset was: partitionUuid = <PARTITION_UUID>, seqno = <SEQ_NO>, snapshot = [X], collectionManifestUid = 0"
for all 1024 of our partitions after we performed a cluster rotation which caused all vBuckets to have a different UUID. This is expected because the UUID changed.
However, we noticed that over time, the “bySeqno” value for all partitions is not increasing. At one instant, the value may be 4000, next instant it may be 8000, next instant it may be 20000, then later it may be 4000. Should the “bySeqno” always be increasing over time (assuming vBucket UUID stay the same and document changes occur all the time)?
Below is an example for one partition (from the offsets):
At time 01:00, the offset for vBucket 781 may look like this:
"[\"nosql_cb3_data1\",{\"bucket\":\"data1\",\"partition\":\"781\"}]" : "{\"vbuuid\":21044398436119,\"collectionsManifestUid\":0,\"snapshotEndSeqno\":3266822,\"bySeqno\":20480,\"snapshotStartSeqno\":0}"
At time 01:05, the offset for the same vBucket may look like this:
"[\"nosql_cb3_data1\",{\"bucket\":\"data1\",\"partition\":\"781\"}]" : "{\"vbuuid\":21044398436119,\"collectionsManifestUid\":0,\"snapshotEndSeqno\":3278968,\"bySeqno\":4096,\"snapshotStartSeqno\":0}"
^ Above we see that the “snapshotEndSeqno” value increases over time, but the “bySeqno” value decreased. We did not observe any error or warning logs during this time (except the warning about rollback for all partitions, but this occurred once per partition). Is it normal for offsets to be like this?
