new tap connection established with no flags set recieves tap_cmd_delete for items that are already expired

Matt Ingenthron added a comment - 07/Jun/11 9:13 AM Need just a little more info. What is the version of software in use?

Keith Batten added a comment - 07/Jun/11 2:00 PM I did a quick test which seems to reproduce this quickly: 1) set 100 keys with 30s expiry 2) wait 3 minutes 3) do a backfill with tap.py and count number of deletes sent (should be 0) results with both 1.6.5.3 and 1.7: curr_items shows 100 items the tap shows 0 adds, 0 mutations, 100 deletes coming over the wire /opt/membase/1.6.3/bin/memcached/memcached -v -c 80000 -p 11211 -E /opt/membase/1.6.3/bin/ep_engine/ep.so -e "expiry_window=0;tap_idle_timeout=10;ht_size=12582917;dbname=/ebs/membase/default;min_data_age=0;queue_age_cap=900;max_size=1\ 3958643712;initfile=/tmp/init.sql;vb0=true" -u membase

Farshid Ghods added a comment - 08/Jun/11 3:56 AM - edited 1- mem_used for expired items benchmarks around delete and expire commands started with 0 items : mem_used 137 MB added 4000 keys each value 10k with key size 40 byte = approximately +412 MB mem_used after setting all the items >600 MB as expected I deleted all those items and mem_used dropped to 200MB start with 0 items : mem_used 137 MB set the same set of keys with expiration set to 1 second mem_used jumps to 600 MB after setting the items but it does not drop down after all items expires. after restarting membase_server the mem_used is back to normal ( 136 MB) 2- stats curr_items : curr_items include the expired items in the curr_items stat so in the above test case I did insert 4000 items with expiry set to 1 and 4 items with expiry set to 0 curr_items : 4004 after restarting membase server the curr_items count is now 0 3- receiving CMD_DELETE over TAP_BACKFILL I do receive a CMD_TAP_DELETE if I connect even after all items are expired. CMD_TAP_DELETE: ``key_c27c31f1-8db5-4977-97c9-468c5e283357_3435'' (vb:989) -> (0 bytes from ('172.16.75.130', 11210))

Farshid Ghods added a comment - 08/Jun/11 4:05 AM possibly dupe of http://www.couchbase.org/issues/browse/MB-3286 which was fixed in 1.6.5

Keith Batten added a comment - 08/Jun/11 1:41 PM By setting exp_pager_stime to 10 (seconds) I'm not seeing CMD_TAP_DELETE on 1.6.5, 1.6.5.1, 1.6.5.3 or 1.7 after waiting for expiration to go through. So it looks like more investigation into the problem is needed.

Farshid Ghods added a comment - 08/Jun/11 2:46 PM ran same set of tests i mentioned in the previous comment ( 4000 items 10k each . expiry set to 10 ) i changed the ep-engine parameter to run the expiry-pager every 10 seconds instead of 1 hours the mem_use increases from 500MB to 600 MB ( expected) after 10 seconds it drops back to 500 MB so we are not able to reproduce the cmd_tap_delete and memory overhead reported here when expiry_pager is set to 10 seconds instead of one hour. I will run the same set of tests against memcached without changing any parameter

Farshid Ghods added a comment - 08/Jun/11 3:23 PM in 1.6.5.3 tap client receives tap_cmd_delete only when the user deletes the key in 1.7.0 tap client receives tap_cmd_delete when user deletes the key or when the item is expired and the expiry pager has not run yet we reproduced the scenario listed in the bug on 1.6.5.3 and unable to see the behavior mentioned in the bug description I will take a look at the burtis script attached.

Farshid Ghods added a comment - 08/Jun/11 3:24 PM reducing the priority to critical for now because it is not 100% reproducible and does not happen all the time while doing a set command with expiry values less than 10 seconds.

Farshid Ghods added a comment - 08/Jun/11 3:37 PM Sriharsha, did you happen to attach the tap client at least one hour after setting those items with expiration=1,3 seconds the expiry pager kicks off every one hour so mem_used will drop only after an hour. but it still doesn't explain why you noticed a "tap_cmd_delete" there in your 1.6.5.3 deployment . tap_cmd_delete is sent for expired items only in 1.7.0 membase.

Farshid Ghods added a comment - 08/Jun/11 3:38 PM I will also try to run brutis script against 1.6.5.3 and 1.7.0 to confim

Farshid Ghods added a comment - 08/Jun/11 5:24 PM latest observation: it doesn't happen if tap client connects with BACKFILL - this happens when client connects with no option - it doesn't explain the high memory usage in this case so they are two separate issues. i looked at the brutis script and implemented a test case very similar. 1- loop over same 4000 keys , each 10k and set the expiration to 1 and 3 2- pager kicks off every 10 seconds 3- memory usage fluctuates between 100-400 MB ( because items do get expired and flushed every 10 seconds)

Chiyoung Seo added a comment - 30/May/12 7:15 PM I don't 'think this is an issue.