couchbase server crashed after run setgettests.MembaseBucket.test_value_100b test

Farshid Ghods added a comment - 09/Jul/12 12:48 PM im confused , i ran some tests on windows 7 64-bit and thye passed last night.

Thuan Nguyen added a comment - 09/Jul/12 12:56 PM Windows 7 and windows 2008 R2 are a little different. Some time it works for windows 2008 but not for windows 7 and vice versa Here the output of view test on windows 2008 with build 2.0.0-1420 http://qa.hq.northscale.net/job/single-node-windows-64-viewtests/34/console

Farshid Ghods added a comment - 09/Jul/12 1:03 PM this run succeeded against windows 7 http://qa.hq.northscale.net/job/single-node-2.0.x-windows7-64-smoke/5/console so clearly things dont work on windows 2008 but work against windows 7

Thuan Nguyen added a comment - 09/Jul/12 1:21 PM Kill memcached process, delete default bucket and its database files manually. Then I can create default bucket but memcached crashed after I delete default bucket. I think after many times create and delete bucket, we could not create bucket any more unless we clean up the system. Port server memcached on node ' ns_1@10.3.2.144 ' exited with status 255. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Extension support isn't implemented in this version of bucket_engine Failed to load mutation log, falling back to key dump metadata loaded in 304 ms warmup completed in 384 ms Shutting down tap connections! Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Extension support isn't implemented in this version of bucket_engine Failed to load mutation log, falling back to key dump metadata loaded in 274 ms warmup completed in 369 ms Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Warning: failed to load the engine session stats due to IO exception "basic_ios::clear" Failed to load mutation log, falling back to key dump Extension support isn't implemented in this version of bucket_engine metadata loaded in 753 usec warmup completed in 1253 usec Shutting down tap connections! Shutting down tap connections! This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.

Aaron Miller added a comment - 09/Jul/12 3:05 PM This doesn't seem to be a 2K8 vs. Win7 issue, as the view and smoke tests pass fine on the win2k8 test build machine 10.3.121.142

Thuan Nguyen added a comment - 09/Jul/12 4:17 PM HIt this bug again in 2.0.0-1421 with clean install in windows 2008 32bit. Here is windows vm in failed state http://10.3.2.179:8091/index.html#sec=buckets Port server memcached on node ' ns_1@10.3.2.179 ' exited with status 255. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Warning: failed to load the engine session stats due to IO exception "basic_ios::clear" Failed to load mutation log, falling back to key dump Extension support isn't implemented in this version of bucket_engine metadata loaded in 949 usec warmup completed in 1671 usec This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.

Thuan Nguyen added a comment - 09/Jul/12 6:35 PM I can repo this bug during smoke test in build 2.0.0-1421 on windows R2 64 bit. Couchbase server will crash if I create any bucket

Aaron Miller added a comment - 09/Jul/12 6:43 PM I really can't get much information as to where its failing since I can't cause this on a machine with dev tools in order to hook in a debugger, or anything of the sort. I am trying to make a build with CouchKVStore patched to use windows-style pathnames, as that's the only thing I could think of that would prevent the files from actually getting created. I'll post it here once it's ready to test.

Thuan Nguyen added a comment - 09/Jul/12 7:34 PM I setup for Aaron running windows smoke test on his dev box. It will help Aaron get more information to trouble shoot the issue.

Aaron Miller added a comment - 10/Jul/12 11:41 AM - edited [snip]

Aaron Miller added a comment - 10/Jul/12 12:14 PM Scratch that, got the issue to happen. Looks less like an issue just with bucket creation and more that something that happens near the end of the smoke test suite causes memcached to freeze up /entirely/ on windows. It's also sitting on quite a bit of memory (1GB with no live buckets).

Aaron Miller added a comment - 10/Jul/12 1:01 PM Looks like at some point creating a bucket wound up hanging memcached. This thread seems be stuck forever: #12 0x0000000071181b96 in EventuallyPersistentEngine::initialize (     this=0x9421200,     config=0x5b2f916 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size= 10000;max_size=1072693248;tap_keepalive=300;dbname=c:/Program Files/Couchbase/Se rver/var/lib/couchbase/data/set-get-bucket-replica-2;allow_dat"...)     at ep_engine.cc:1352 #13 0x00000000711823c6 in EvpInitialize (handle=<value optimized out>,     config_str=0x5b2f916 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_s ize=10000;max_size=1072693248;tap_keepalive=300;dbname=c:/Program Files/Couchbas e/Server/var/lib/couchbase/data/set-get-bucket-replica-2;allow_dat"...)     at ep_engine.cc:117 #14 0x000000006bd0314d in create_bucket_UNLOCKED (e=0x6bd0d000,     bucket_name=0x5b2fb20 "set-get-bucket-replica-2",     path=0x5b2f8e0 "c:/Program Files/Couchbase/Server/lib/memcached/ep.so",     config=0x5b2f916 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size= 10000;max_size=1072693248;tap_keepalive=300;dbname=c:/Program Files/Couchbase/Se rver/var/lib/couchbase/data/set-get-bucket-replica-2;allow_dat"...,     e_out=0x0, msg=0x5b2f4d0 "", msglen=1024) at bucket_engine.c:705 #15 0x000000006bd0333d in handle_create_bucket (handle=0x6bd0d000,     cookie=0x57cc000, request=<value optimized out>,     response=0x4042f0 <binary_response_handler>) at bucket_engine.c:2141 #16 0x000000006bd03ac8 in bucket_unknown_command (handle=0x6bd0d000,     cookie=0x57cc000, request=0x57e6800,     response=0x4042f0 <binary_response_handler>) at bucket_engine.c:2407 #17 0x00000000004105ac in process_bin_unknown_packet (c=0x57cc000)     at daemon/memcached.c:2878 #18 process_bin_packet (c=0x57cc000) at daemon/memcached.c:3166 #19 complete_nread_binary (c=0x57cc000) at daemon/memcached.c:3737 #20 complete_nread (c=0x57cc000) at daemon/memcached.c:3819 #21 conn_nread (c=0x57cc000) at daemon/memcached.c:5667 #22 0x0000000000404ee5 in event_handler (fd=<value optimized out>,     which=<value optimized out>, arg=0x57cc000) at daemon/memcached.c:5930 #23 0x0000000065984695 in event_process_active_single_queue (base=0x5824500,     flags=<value optimized out>) at event.c:1308 #24 event_process_active (base=0x5824500, flags=<value optimized out>)     at event.c:1375 #25 event_base_loop (base=0x5824500, flags=<value optimized out>)     at event.c:1572 #26 0x00000000004131b1 in worker_libevent (arg=0x5834000)     at daemon/thread.c:301 #27 0x0000000062484e4e in pthread_timechange_handler_np ()    from c:\Program Files\Couchbase\Server\bin\pthreadGC2.dll #28 0x00000000011742a0 in ?? () #29 0x000000006ea3394c in ___DTOR_LIST__ ()    from c:\Program Files\Couchbase\Server\bin\libtcmalloc_minimal-4.dll #30 0x0000000000000000 in ?? ()

Aaron Miller added a comment - 10/Jul/12 1:05 PM Also, I can reproduce this issue quickly with -just- the test "setgettests.MembaseBucket.test_value_100b".

Farshid Ghods added a comment - 10/Jul/12 1:10 PM should we assign tthis to couchbase bucket team ?

Aaron Miller added a comment - 10/Jul/12 1:41 PM Probably should be assigned to someone who can better investigate memcached/ep-engine issues, yes.

Thuan Nguyen added a comment - 10/Jul/12 1:56 PM I alos see this bug hit windows 7 pro 64 bit with "setgettests.MembaseBucket.test_value_100b" test

Jin Lim added a comment - 11/Jul/12 12:28 AM The root cause of this issue is the same as MB-5689 . Basically during the automated test any attempts to delete bucket would fail if the data directory still contains stats.json & access.log. Unfortunately, consequent warmup or creation of a bucket with the same name would also fail for an invalid access log(database directory is empty but the access log still holding old deleted items). We should fix the issue, MB-5689 , right away before running any other regression test. If not, at least manually delete all Couchbase data files before launching a new test. Thanks!