[MB-4513] ep-engine fails with assertion еrror during/right-after warmup Created: 07/Dec/11  Updated: 10/Apr/12  Resolved: 10/Dec/11

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.0-developer-preview-3
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Aleksey Kondratenko Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Start couchbase server. 1 node is enough.

Load few items. (2 was enough here).

Stop it. Start it back.

Observe how memcached will repeatedly crash.

With the following in core dump:

Program terminated with signal 6, Aborted.
#0 0xf76f7430 in __kernel_vsyscall ()
(gdb) up
#1 0xf750b911 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
#2 0xf750ed42 in *__GI_abort () at abort.c:92
92 abort.c: No such file or directory.
in abort.c
#3 0xf7504b28 in *__GI___assert_fail (assertion=0xf3c5e49c "ht.cacheSize.get() < ((size_t)1<<(sizeof(size_t)*8-1))", file=0xf3c5e3df "stored-value.cc", line=377,
    function=0xf3c5e6a0 "static void StoredValue::reduceCacheSize(HashTable&, size_t, bool)") at assert.c:81
81 assert.c: No such file or directory.
in assert.c
#4 0xf3afb4fd in StoredValue::reduceCacheSize (residentOnly=<optimized out>, by=<optimized out>, ht=<optimized out>) at stored-value.cc:377
377 assert(ht.cacheSize.get() < GIGANTOR);
#5 0xf3b70e64 in StoredValue::reduceCacheSize (ht=..., by=91, residentOnly=false) at stored-value.cc:377
377 assert(ht.cacheSize.get() < GIGANTOR);
(gdb) p ht.cacheSize.value
$1 = 4294967292
(gdb) p/x ht.cacheSize.value
$2 = 0xfffffffc

Comment by Trond Norbye [ 07/Dec/11 ]
You've done some stuff around cache size lately? does this look familiar?
Comment by Aleksey Kondratenko [ 07/Dec/11 ]
BTW, I've tried rewinding ep-engine few commits (we made 1.8 merge recently) without success.
Comment by Mike Wiederhold [ 07/Dec/11 ]
Yeah, I have been doing work in this area, but the big change I made hasn't gotten into 1.8 yet. My plan is to merge it into 1.8.1 when we have more resources for QA since I don't want to create issue like these.
Comment by Keith Batten (Inactive) [ 08/Dec/11 ]
centos 5.4 64bit
1 node

1) create default bucket 200M
2) load in 2 items
3) run "service couchbase-server restart"
4) watch the logs/processes to see memcached repeatedly failing

its possible this only happens at low item counts. running with 100k and 10M items did not repro this, while 2 items repeatedly reproduced it
Comment by Dipti Borkar [ 09/Dec/11 ]
Mike, is this merged?
Comment by Thuan Nguyen [ 29/Dec/11 ]
Integrated in github-ep-engine-2-0 #131 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/131/])
Generated at Fri Apr 18 01:53:47 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.