set rejection (write error) logging

The log itself does not explicitly print these messages, but you may be able to monitor the process-level statistics to get more info.

-For when a server is down, obviously you won't be able to get stats from it. You can either monitor it with your own monitoring system (cacti, nagios, etc) or monitor the Membase UI or REST API to get the node statuses
-I'm not sure I understand when the "set rate is too high". Is this related to the fact that the memory is growing so quickly that it can't write to disk fast enough to make more room? In this case, you'll see the "tmp_oom" statistic grow:
/opt/membase/bin/ep_engine/management/stats (for 1.6.x) or /opt/membase/bin/mbstats (for 1.7):
ep_tmp_oom_errors: 0

-When the memory is SO full that there's no way to make room for more data (usually when metadata is filling up all the RAM), you'll get a "hard" oom and you can monitor that stat via:
/opt/membase/bin/ep_engine/management/stats (for 1.6.x) or /opt/membase/bin/mbstats (for 1.7):
ep_oom_errors: 0

Hope that helps

perry

I’m trying to stress test a membase cluster composed by 3 nodes with 600 MB of RAM each.
I do not understand why I get “set failed” errors after a few seconds of running my test scripts. I noticed that the script aborts when the memory shown to be used by the membase control panel reaches 65% of the total 600MB assigned.
If I wait a few minutes after the set failures, I see some of the membase memory is released and if I run my script again it run successfully for a few seconds before there is a new failure to set a value.

Here is my script:

#!/usr/bin/perl -w
use Cache::Memcached::libmemcached;
die "args: srv:puerto inicio cant accion\n\naccion: put|get|del\n" if (scalar @ARGV < 3 );
my $servport=$ARGV[0];

my $memd = Cache::Memcached::libmemcached->new(
{servers => [$servport]} );
die ("couldn't connect $servport") if not defined $memd;

my $start = $ARGV[1];
my $cant = $ARGV[2];
my $act = $ARGV[3];

&stress($start,$cant);

sub stress
{
my $num = shift;
my $cant = shift;
while ($num++ < $cant)
{
&$act($num);
}
}

sub put
{
my $num= shift;
$memd->set("$num" ,"x" x 1000000 .localtime().':'.time()) or die "couldn't set $num";
print "INSERT/UPDATE $num\n";
}

sub del
{
my $num= shift;
$memd->delete("$num") or warn "couldn't delete $num";
print "DELETE $num\n";
}

sub get
{
my $key=shift;
my $val = $memd->get($key) ;
$val = substr $val, -16, 20;
$val="undef" if not defined $val;
print "$key:$val\n"
}

Here is my output when I run it:

~# ./membase.pl 10.4.0.64:11212 0 5000 put
INSERT/UPDATE 1
INSERT/UPDATE 2
...
INSERT/UPDATE 557
INSERT/UPDATE 558
couldn't set 559 at ./membase.pl line 58.

The error in web console: Hard Out Of Memory Error. Bucket "test2" on node 10.4.0.64 is full. All memory allocated to this bucket is used for metadata

The bucket initially was empty

~# /opt/membase/bin/mbstats 10.4.0.66:11212 all |grep oom
ep_oom_errors: 1
ep_tmp_oom_errors: 2874
ep_warmup_oom: 0

the 2874 number is the before test executed but I delete all data and key from the bucket before re run this script(I don’t know how to reset this values)

machine configuration (Virtual machines over VMWare)
~# cat /etc/issue
Debian GNU/Linux 5.0 \n \l

~# uname -a
Linux v171ucert 2.6.26-2-amd64 #1 SMP Thu Sep 16 15:56:38 UTC 2010 x86_64 GNU/Linux

top - 14:51:50 up 90 days, 21:58, 1 user, load average: 0.97, 0.97, 0.50
Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
Cpu0 : 1.3%us, 0.2%sy, 0.0%ni, 98.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 1.1%us, 0.2%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8202892k total, 8149056k used, 53836k free, 64864k buffers
Swap: 1331192k total, 596k used, 1330596k free, 5245732k cached

Let me try to describe what appears to be happening here.

I'm not a perl expert, but it looks like your code is basically just doing as many sets as possible until one fails. Please correct me if I am wrong here. If this is the case you begin doing a buch of sets and they succeed for a little bit, but all of a sudden you get a set that fails. This failure is because you are writing data to Membase faster than it can write data to disk so at some point you will end up filling memory with data that is waiting to be persisted to disk and meta data. (Meta data is stored for each key in Membase and basically tells where the value resides be it on disk or in memory and how to get that value). When memory is in this condition and you try to do another set Membase will return to you an error for that set saying "Temporarily Out of Memory" or Temp OOM. What Membase is telling you here is that it is temporarily unavailable and to try again later and everything should work just fine. This is why when your program stops and you run it again it runs for a little bit before it gets another Temp OOM error. I'll bet too that each time you run the program without flushing the buckets data that it runs for a shorter and shorter period of time. This happens because the amount of meta data is increasing as you add new items so the amount of room that new data has to sit in memory before being written to disk is gradually decreasing. That error message you were seeing in the Membase web ui that said memory was 65% full didn't mean that you were only using 390MB of memory, it meant that your meta data was taking up 390MB of memory. This warning is basically telling you that your meta data is taking up so much space in you cluster that you are starting to get into a bad state. The way to solve this problem is to add more memory or another node to you cluster and if you were running Membase in production and saw this message then you it should be treated as a serious issue.

Eventually when you have done enough sets you see this message:

The error in web console: Hard Out Of Memory Error. Bucket "test2" on node 10.4.0.64 is full. All memory allocated to this bucket is used for metadata

This means meta data has take up so much memory that you cannot do any more sets or add any more data to your cluster.

So what can you do about this? First of all you can increase your bucket size. This will technically just make the server take longer for you to see any Temp OOM errors or have all your memory taken up by meta data, but in a production setting you would never use a bucket as small as 600MB if you were going to have a lot of keys or if you were going to be bulk loading large amounts of data very quickly into Membase. For what you are trying to do though I think having a small bucket size to test with is good so that you can make sure your client can handle getting Temp OOM errors. The second thing you should do no matter how big or small the bucket is is to implement an exponential backoff when you see a Temp OOM error. Usually if you stop sending things to Membase for even 10ms it will free up a good amount of memory and you will be able to send a bunch more set operations before possibly getting another Temp OOM.

As for the OOM error you get when you fill memory with meta data. This is really bad to get and any cluster can run into this issue. You need to make sure when you do go into production that you have allocated enough resources for Membase to handle all of you data.

Also,

The error in web console: Hard Out Of Memory Error. Bucket "test2" on node 10.4.0.64 is full. All memory allocated to this bucket is used for metadata

The bucket initially was empty
~# /opt/membase/bin/mbstats 10.4.0.66:11212 all |grep oom
ep_oom_errors: 1
ep_tmp_oom_errors: 2874
ep_warmup_oom: 0

the 2874 number is the before test executed but I delete all data and key from the bucket before re run this script(I don’t know how to reset this values)

It looks like this time you actually ran out of memory all together. The above error statement incremented ep_oom_errors and not ep_tmp_oom_errors.

Thank you Mike for the through explanation!
One last question: is it possible to configure membase in a way that it would block the client when it has too many memory pending to be flushed to disk, instead of rejecting the set as it did in our test? if its not possible right know, do you think it's a good idea for a future configuration option in membase?

Best regards

Right now there is not a feature to do this and it is not likely we will add it. A while back there was a lot of discussion about whether or not we should implement this and we decided against it. The reason is that if we had a feature to block on Temp OOM then we wouldn't be slowing down the rate at which the requests were being sent to the server by the client. Imagine if your test continued sending requests after the Temp OOM and the server accepted them. You would be gradually increasing the time it took to get a response because messages would be queueing up on the server-side waiting for the server to have memory to process them. The correct behavior here is for the client to slow it's rate of sending messages and blocking on the server side won't achieve this behavior.

One thing that is a possibility is to have the server delay the response time for requests when the server is approaching a state where a Temp OOM will occur. This would cause Temp OOM to be very unlikely and have the effect of slowing down the client's request rate. On the other hand it is somewhat complex to add to the server and easy to add to the client, hence it is a low priority at the moment. I added an improvement for this to Jira hwever, MB-4020.

I don't know if there is an easy way for the client (using a memcached client api) to know when the server is approaching a Temp OOM so it should slow down by itself. It will only know that it reached the limit on the first set reject, which may have multiple causes, such a node down or unreachable, and not only temp OOM.

Maybe a feature useful for this (and for monitoring) would be to have "system status keys" inside the cache, with specific names (that hopefully won't collision with the keys that the application would use), that any client can query using the memcached api and that will report the percentage of memory used and other variables about the status of the server that a client (or a monitoring service) may need to use. It would be something comparable to what the /proc filesystem is for in linux.

Thanks again for the help!

Well, your client can actually tell the difference between failures. If (and only if) you get the "temp OOM" back, then you know it is time to start slowing down. Other errors have different messages so you can tell the difference.

Most clients also have the ability to gather stats from each (or all) nodes of a cluster and so you can use that to determine how close you are to the limits.

Perry