Issues with Query Consistency in Eventing Functions

[bilalkilic]

We’re using eventing functions to update user notification counts in a destination bucket based on changes in a source bucket. Our query:

SELECT * FROM user_notification_counts WHERE userId = userId;

which uses the index:

CREATE INDEX `user_notification_by_userId` ON `user_notification_counts`(`userId`)

Despite setting N1QL Consistency to request and introducing delays before querying, we still face discrepancies between the source and destination counts. Any insights on improving consistency?

[mreiche]

Despite setting N1QL Consistency to request

Can you please show your eventing function?
And what does it have the discrepancy with?

@bilalkilic

SELECT * FROM user_notification_counts WHERE userId = userId;

That query is going to give all the documents that have a userId property.
How are you counting the “changes”?

[bilalkilic]

Sorry for the confusion this is the function we use

function OnUpdate(doc, meta) {
    let userId = parseInt(meta.id.split("-")[0]);
    var count = SELECT count(1) as count FROM user_notification_counts where userId = $userId;
    for (var c of count) {
        if (c.count && c.count> 0 ) {
            destination_bucket[userId] = c
            log("Doc created/updated", userId);
        }
        else{
            delete destination_bucket[userId]
            log("Doc deleted", userId);
        }
    }
    count.close()
}

function OnDelete(meta, options) {
    let userId = parseInt(meta.id.split("-")[0]);
    var count = SELECT count(1) as count FROM user_notification_counts where userId = $userId;
    for (var c of count) {
        if (c.count && c.count > 0 ) {
            destination_bucket[userId] = c
            log("Doc created/updated", userId);
        }
        else{
            delete destination_bucket[userId]
            log("Doc deleted", userId);
        }
    }
    count.close()
}

[mreiche]

And what’s the discrepancy?

[bilalkilic]

When we compare data in source and destination buckets notification counts are different for a large number of documents (approx 1M in 12M documents),

{
  "destination_bucket": {
    "count": 13
  },
  "source_bucket": {
    "count": 14
  }
}

We think the query returns the stale counts before the change in the source bucket is applied. We saw no changes after setting the N1QL Consistency to Request in function settings

[mreiche]

I’m not an eventing expert. How did you set the QueryScanConsistency for the n1ql in the eventing functions?

How did you set the QueryScanConsistency for the query that is returning what you are showing above with the source count=14 and destination count=13?

And why is there an expectation that there be the same documents in the source and destination?

[bilalkilic]

I set the N1QL Consistency in the function settings as stated in the documentation.

The destination bucket is the projection of the source bucket. It is updated with the latest count from the query every time the source bucket is updated.

[mreiche]

Ok, and…

How did you set the QueryScanConsistency for the query that is returning what you are showing above with the source count=14 and destination count=13?