beam.smp high CPU and Memory usage

Hello,

Could you please give more information about your environment?
- which OS?
- do you have the sample buckets installed?
- in your buckets do you have views?

Couchbase 2.0 is consuming more resources than earlier released because of all the activity happening in the background (check status of document on the disk, indexing, ...) but I am surprised by the numbers you are showing.

This blog post explains what is happening in the background :
http://blog.couchbase.com/how-many-nodes-part-1-introduction-sizing-couc...

Also most of the project I have seen so far do not use that many buckets, since the background processes are managed by buckets the less bucket you have, less ressource are consumed (most of the time)

Regards

Especially, let us know if this is windows. Or if you see errors log growing like mad. 450% of CPU usage is indeed very unexpected behavior.

Both nodes are running CentOS 6.3 minimal install on Xen virtualisation, running single core, 1.5G ram.

No sample buckets installed, 8 buckets, averaging 1 view per bucket, half the buckets with no documents, others with 2-5 documents.

It just seems very odd that it's having problems with 1 user, and dozen documents.

I can try to work around with less buckets, But I thought that's the whole point of allocating say.. 100Mb per bucket, so that this doesn't happen?

Ok.

Please check your error logs. I can only explain such high cpu usage by some constant stream of errors.

Feel free to file bug http://www.couchbase.com/issues/secure/Dashboard.jspa and attach cbcollect_info output.

Hi,

I tried to run cbcollect_info,

It had quite a few errors, are these important ?

[root@db01 ~]# /opt/couchbase/bin/cbcollect_info cblog
uname (uname -a) -  OK
Directory structure (ls -lR '/opt/couchbase' /opt/membase /var/membase /etc/opt/membase) -  Exit code 2
Database directory structure (ls -lR /opt/couchbase/var/lib/couchbase/data) -  OK
Directory structure membase - previous versions (ls -lR /opt/membase /var/membase /var/opt/membase /etc/opt/membase) -  Exit code 2
Process list snapshot (top -H -n 1) -  OK
Process list  (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) -  OK
Swap configuration (free -t) -  OK
Swap configuration (swapon -s) -  OK
Kernel modules (lsmod) -  OK
Distro version (cat /etc/redhat-release) -  OK
Distro version (lsb_release -a) -  Exit code 127
Installed software (rpm -qa) -  OK
Installed software (COLUMNS=300 dpkg -l) -  Exit code 127
Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) -  Exit code 127
Process usage (export TERM=linux; top -b -n1 | egrep 'moxi|memcached|vbucketmigrator|CPU|load|Mem:|Swap:|Cpu(s)') -  OK
Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') -  OK
netstat -nap (netstat -nap) -  OK
relevant lsof output (lsof -n | grep 'moxi\|memcached\|vbucketmigrator\|beam') -  Exit code 1
Network configuration (ifconfig -a) -  OK

And my zip file is 62.1 MB, unable to upload to jira with 50MB limit..

I'm finding that the nodes that have high cpu are those with less than a 90% active in memory value. Hope this helps.

The nodes actually became so busy, it didn't respond, and turned pending / down intermittently. This was triggered with a curl REST API update on view.

The view was really simple

function (doc, meta) {
    if (doc.facebook_id)
        emit(doc.facebook_id, null);
    if (doc.email)
        emit(doc.email, null);
    if (doc.twitter)
        emit(doc.twitter, null);
}

But the view update, seemed to tick off the high CPU and high memory, and made it not go south completely.

I have compiled the cbcollect_info zip, but it's 62Mb, do you guys have a suggestion where I can attach this?

Thanks,
Johnson

I suggest you to recompress this data (it's .zip), e.g. with bzip2 and attach to ticket.

Alternatively you can use any free file sharing service.

I'm also having high memory usage (9GB) on view indexing. Was there an issue created for this?

I cannot be sure I understand 9GB. Feel free to file a ticket on that as well.

Is that 9GB of resident memory usage of beam.smp ?

Not sure what resident memory usage means. I checked using 'top' and oom killer came by to kill beam.smp.

Ok. Looks like genuine problem. Please file a bug.

Do you use spatial views btw ? Or they're all "plain" ?

No just two plain views. They do emit quite large documents into the index.

ok. How large?

In general current views implementation doesn't really likes bigger values. Only bigger keys are worse.