@david.nault, Essentially i need a way to get the latest of the documents in couch i.e. a document with key 100 may have multiple events (t1,t2,…,t10) and i ingest these events using Kafka connector. I should be able to derive the latest which is t10 out of these 10 events (my questions in the other thread is related to this).
Determining the latest of every document using the application logic is quite complicated and hence resorting to a technical solution. I can use the bySeqNo but as you mentioned in the other thread, its an issue during the failover scenario (Ofcourse, i need to check the persistence polling that you mentioned)
Keeping the persistence polling aside for now…
So i have written a connect filter and in the poll method, i add a epoch timestamp to the source record and use this timestamp to determine the latest (as the events in the same vBucket would go to the same task and in order, so adding a timestamp would help me determine the latest). The problem that i am facing now is that the same timestamp is added to different events, i guess kafka connector processes more events per millisec.
Now i am thinking of adding a counter also apart from the timestamp. The counter is an instance variable in the filter and its incremented for every event. Since the events in a vBucket go to the same connector task and in order, the counter that i add would also be incremental. I compare the timestamp first and then compare the counter only if the timestamp is the same. I have to do this keeping the failure scenario in mind i.e. if the kafka connector worker or task or node fails, then the counter would be reset and comparing the counter first would not help. In this case, the timestamp would definitely differ as it would take more than a millisec to recover but from then on, the counter would help even if the timestamp is the same. I guess keeping the synchronized static counter (AtomicLong) would also help.
On the point that you made - " If you’re set on this path, I’d recommend using a static AtomicLong. But as I’m sure you’re aware, using an in-memory counter would limit you to running a single connect worker."
As the events in the same Vbucket go to the same connector task and i want the ordering per document (document with the same key go to the same vBucket), this should not be an issue even if i have multiple workers. I do not want a global counter but the counter which would help me per document events.
What are your thoughts?