items are not marked as deleted/expired in couchstore after they expire (View query results with stale=false include expired items)

Filipe Manana added a comment - 14/Aug/12 10:35 AM That's not an expected. Items are lazily expired by ep-engine, meaning that it will not perform document deletes in the database after the 30 seconds. There's no way to control that or know that from the view-engine.

Farshid Ghods added a comment - 14/Aug/12 10:55 AM seems like something can be modified in ep-engine so that when items expire we dont see them in views anymore

Peter Wansch added a comment - 14/Aug/12 11:16 AM Chiyoung, is this something Jin or Mike can help out with if it's in ep_engine? If not, it may need to be passed to Aaron. Thank you.

Filipe Manana added a comment - 14/Aug/12 11:24 AM This was discussed internally a few times, but I don't think any decision was made. Mike gave some info in the forum to a user about this: http://www.couchbase.com/forums/thread/expiration-time-docs-dp4

Chiyoung Seo added a comment - 14/Aug/12 12:08 PM The item or expiry pager wasn't scheduled yet to clear up all expired items from memory hashtable and disk. That's why you still see those expired items in the view query. The item pager will be scheduled if the current memory usage is above high water mark. The expiry pager will be scheduled once every hour by default, but you can change the expiry pager's interval to a shorter period (e.g., 5 minutes) at runtime.

Farshid Ghods added a comment - 14/Aug/12 12:27 PM Dipti, this means that users will see the expired items in the index for sometimes up to an hour which is the default value for the expiry pager.

Dipti Borkar added a comment - 14/Aug/12 12:48 PM Peter, as discussed this is something we should be able to do at query time. We do need to fix this for 2.0. Can you please help understand the options with the view engine team? Let me know if you need additional feedback from me.

Filipe Manana added a comment - 14/Aug/12 2:34 PM There's no efficient way to do this in view engine. It would imply for each stale=false request to scan all documents in every vbucket and check if they expired, not to mention other smaller issues.

Peter Wansch added a comment - 15/Aug/12 10:17 AM Deep, can you confirm that after an hour, once the expiry pager has run and the next time the indexes are updated, they disappear from the view? If so, then we don't have a bug. There is still a valid discussion going on about how the situation around queries can be improved but I want to find out if things are working as designed for now.

Deepkaran Salooja added a comment - 16/Aug/12 10:12 AM Yes, that's correct. Once the expiry pager has run and indexes have been updated, the queries do not return the expired items.

Perry Krug added a comment - 22/Aug/12 6:48 AM Just a thought as I came across this bug. What if for each query result, the query engine contacted memcached to see if each doc was still valid before including it in the query response? That way, the view engine wouldn't have to keep track of all documents in all vbuckets, only the ones that it is sending out. This would not only take care of expiration (since memcached would return "not_found") but also deleted documents that have not yet been removed from disk. Rather than doing a 'get' (which would fetch it from disk in DGM), we could use the "stats key" operation to just check whether the key is still valid within memcached. Since there would be a bit (ableit small) amount of overhead on the query response, this could be an optional check? The rows would eventually get cleaned from the index, this is just preventing the client from getting a massive amount of already expired items at the minute 59 mark before the hourly process is run.

Filipe Manana added a comment - 22/Aug/12 7:04 AM Thanks for the suggestion Perry. Unfortunately it wouldn't work for several reasons. First view-engine has no way to communicate with memcached currently. Second, it would slow things down significantly. Third, how could that work for reduces? For precomputed reduce values, which is the strength of couchdb's btrees + mapreduce, how do you "unreduce", exclude values produced for expired documents, and re-compute reductions? Not only you would need to know the map values produced by expired documents, you would also need to know the map values for the non-expired documents. Not to mention the big performance penalty here. There's a lot of other technical issues that would impact either correctness, the incremental view update approach or performance. Those 3 listed above are just the ones people in general not familiar with implementation/design would grasp quickly.

Peter Wansch added a comment - 05/Sep/12 12:21 PM Filipe explained in the comments how this works. To speed up deletion from the indexes, the expiry pager interval can be changed which may have an adverse effect in performance.