http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6219 Couchbase Server <br/> View query results with stale=false include expired items. <br/> <br/> Steps to reproduce(build#1580): <br/> 1. Create default bucket <br/> 2. Load 10 json docs with expiry set to 30 seconds. <br/> 3. Create a view(default map func) and query with stale=false. <br/> 4. Wait for 2-3 minutes. <br/> 5. Query view again with stale=false. <br/> <br/> Some of the items are still returned in the query results even when index is rebuilt. <br/> I observed that the number of rows returned by the view query is always the same as curr_items. <br/> <br/> Diagnostics are attached. <br/> <br/> build#1580 on Ubuntu 64bit MB-6219

items are not marked as deleted/expired in couchstore after they expire (View query results with stale=false include expired items)

Bug Major Closed Won't Fix Public Peter Wansch Deepkaran Salooja Tue, 14 Aug 2012 10:32:01 -0500 Wed, 26 Sep 2012 19:37:32 -0500 Wed, 5 Sep 2012 12:21:45 -0500 recent-builds-2.0 2.0 couchbase-bucket view-engine 0 4 That&#39;s not an expected.<br/> Items are lazily expired by ep-engine, meaning that it will not perform document deletes in the database after the 30 seconds.<br/> <br/> There&#39;s no way to control that or know that from the view-engine. seems like something can be modified in ep-engine so that when items expire we dont see them in views anymore Chiyoung, is this something Jin or Mike can help out with if it&#39;s in ep_engine? If not, it may need to be passed to Aaron. Thank you. This was discussed internally a few times, but I don&#39;t think any decision was made. <br/> <br/> Mike gave some info in the forum to a user about this: <br/> <br/> <a href="http://www.couchbase.com/forums/thread/expiration-time-docs-dp4">http://www.couchbase.com/forums/thread/expiration-time-docs-dp4</a> <br/> <br/> The item or expiry pager wasn&#39;t scheduled yet to clear up all expired items from memory hashtable and disk. That&#39;s why you still see those expired items in the view query.<br/> <br/> The item pager will be scheduled if the current memory usage is above high water mark. The expiry pager will be scheduled once every hour by default, but you can change the expiry pager&#39;s interval to a shorter period (e.g., 5 minutes) at runtime. Dipti,<br/> <br/> this means that users will see the expired items in the index for sometimes up to an hour which is the default value for the expiry pager. Peter, as discussed this is something we should be able to do at query time. We do need to fix this for 2.0. Can you please help understand the options with the view engine team? <br/> Let me know if you need additional feedback from me. There&#39;s no efficient way to do this in view engine. It would imply for each stale=false request to scan all documents in every vbucket and check if they expired, not to mention other smaller issues. Deep, can you confirm that after an hour, once the expiry pager has run and the next time the indexes are updated, they disappear from the view? If so, then we don&#39;t have a bug. There is still a valid discussion going on about how the situation around queries can be improved but I want to find out if things are working as designed for now. Yes, that&#39;s correct. Once the expiry pager has run and indexes have been updated, the queries do not return the expired items. Just a thought as I came across this bug. What if for each query result, the query engine contacted memcached to see if each doc was still valid before including it in the query response? That way, the view engine wouldn&#39;t have to keep track of all documents in all vbuckets, only the ones that it is sending out. This would not only take care of expiration (since memcached would return &quot;not_found&quot;) but also deleted documents that have not yet been removed from disk. Rather than doing a &#39;get&#39; (which would fetch it from disk in DGM), we could use the &quot;stats key&quot; operation to just check whether the key is still valid within memcached. Since there would be a bit (ableit small) amount of overhead on the query response, this could be an optional check? <br/> <br/> The rows would eventually get cleaned from the index, this is just preventing the client from getting a massive amount of already expired items at the minute 59 mark before the hourly process is run. Thanks for the suggestion Perry. <br/> Unfortunately it wouldn&#39;t work for several reasons. <br/> <br/> First view-engine has no way to communicate with memcached currently. <br/> <br/> Second, it would slow things down significantly. <br/> <br/> Third, how could that work for reduces? For precomputed reduce values, which is the strength of couchdb&#39;s btrees + mapreduce, how do you &quot;unreduce&quot;, exclude values produced for expired documents, and re-compute reductions? Not only you would need to know the map values produced by expired documents, you would also need to know the map values for the non-expired documents. Not to mention the big performance penalty here. <br/> <br/> There&#39;s a lot of other technical issues that would impact either correctness, the incremental view update approach or performance. Those 3 listed above are just the ones people in general not familiar with implementation/design would grasp quickly. Filipe explained in the comments how this works. To speed up deletion from the indexes, the expiry pager interval can be changed which may have an adverse effect in performance. Date of First Response Tue, 14 Aug 2012 10:35:05 -0500 Rank 4362 Sprint Priority 5.0 Time In Status