http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-4394 Couchbase Server create a cluster of 10 node couchbase server 2.0.0r-177 <br/> Load 30 million items to cluster so that reach about 85% resident. <br/> Keep the load runing and add 5 more nodes 2.0.0r-177 <br/> Rebalance cluster. Failed. centos 5.4 64 bit on ecs MB-4394

memcached crash while rebalancing 15 nodes with 30M items (FATAL: Object returned from mccouch with CAS == 0)

Bug Critical Closed Duplicate Public Dustin Sallings Thuan Nguyen Mon, 31 Oct 2011 12:36:44 -0500 Thu, 10 Jan 2013 01:21:34 -0600 Mon, 7 Nov 2011 22:59:19 -0600 2.0 2.0 bucket-engine 0 1 Port server memcached on node &#39;<a href='mailto:ns_1@10.34.149.233'>ns_1@10.34.149.233</a>&#39; exited with status 134. Restarting. Messages: Preloaded 5898290 keys (with metadata) <br/> tcmalloc: large alloc 4294938624 bytes == 0x49c5e000 @ <br/> FATAL: Object returned from mccouch with CAS == 0 Found myself commenting on a bug in email. Need to be careful about that. <br/> <br/> <br/> <br/> The error is misleading. It appears to be one of these two things: <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;static bool decodeMeta(const uint8_t *dta, uint32_t &amp;seqno, uint64_t &amp;cas, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;uint32_t &amp;length, uint32_t &amp;flags) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (*dta != 0x01) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Unsupported meta tag <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;++dta; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (*dta != 20) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Unsupported size <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} <br/> <br/> Considering the prior allocation was 4GB, I&#39;m guessing that something read something incorrectly and we&#39;re just off by this point. <br/> <br/> Any chance this is a small database I can play with? <br/> I didn&#39;t mean to assign this to myself, but I&#39;m going to pass a baton briefly to Farshid for some reproduction data.<br/> <br/> I&#39;d *really* like an attachment shard that does this so I can try to do the same thing in isolation. Although this is also very interesting from one of the attached stacks. How many different things are going wrong here? <br/> <br/> <br/> <br/> Thread 1 (Thread 0x7f8e0c97e700 (LWP 22692)): <br/> #0 0x0000000000000000 in ?? () <br/> #1 0x00007f8e0d294b1c in Task::maxExpectedDuration (this=0x4b2c3000) at dispatcher.hh:152 <br/> #2 0x00007f8e0d2945a2 in Dispatcher::run (this=0xf5b9000) at dispatcher.cc:136 <br/> #3 0x00007f8e0d29479c in launch_dispatcher_thread (arg=0xf5b9000) at dispatcher.cc:28 <br/> #4 0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f8e11863ead in clone () from /lib64/libc.so.6 <br/> can&#39;t access those vms anymore. they seem to be terminated by now.<br/> <br/> Tony,<br/> <br/> can you please provide the ratio of set/get/delete/expire which you run the mixloader with ? and possibly copy-paste that part of the mix-loader which loops over all the keys and run those memcached commands I just got a crash again today and have 3 new core attached in here. <br/> I run 15 python threads to load 30 millions items. The script does all set. <br/> python scripts/mixload-allset.py -i manual2.0.ini -p prefix=key_01,size=655,count=2000000 &amp; <br/> Command to run memcachetest. <br/> ./memcachetest -h 184.72.85.127:11211 -i 100000 -c 50000 -m 128 -t 2 -l&#xA0; <br/> After finish loading 30 million items, I do 70% set,get, 30% delete and 30% set again. <br/> counter_10 = 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;all_set = False <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;while i &lt; count: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;key = &quot;{0}-{1}&quot;.format(prefix, i) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if counter_10 &gt;= 7: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if all_set == True: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mc.delete(key) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mc.set(key, 0, 0, payload) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if counter_10 == 10: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;counter_10 = 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mc.set(key, 0, 0, payload) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;mc.get(key) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;counter_10 += 1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;i += 1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if i == int(count): <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;all_set = True <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;i = 0 <br/> looked at the more recent core logs <br/> its a dupe of <a href="http://www.couchbase.org/issues/browse/MB-4412">http://www.couchbase.org/issues/browse/MB-4412</a> <a href="http://www.couchbase.org/issues/browse/MB-4412">http://www.couchbase.org/issues/browse/MB-4412</a> Date of First Response Mon, 31 Oct 2011 13:23:24 -0500 Rank 6046 Time In Status