Low Couchbase RAM Utilization

Couchbase uses a concept of high and low watermarks that are set at 80% and 65% respectively. If you continue to add data or do get requests where the item happens to be on disk then you will see that you memory usage will increase up to 86 GB. At this point the item pager will kick in and remove 16GB or data using an LRU like algorithm. If your cluster is under really heavy load then you will see the memory usage go up to around 100GB.

As you pointed out the high and low watermarks cause memory to be unused during periods of time and in future releases of Couchbase we will remove them.

If you would like to modify these watermarks then you can change them with

Depending on your release you will use one of these scripts

./cbepctl localhost:11210 set flush_param mem_high_wat <value in bytes>
./cbepctl localhost:11210 set flush_param mem_low_wat <value in bytes>

or

./cbflushctl localhost:11210 set flush_param mem_high_wat <value in bytes>
./cbflushctl localhost:11210 set flush_param mem_low_wat <value in bytes>

Thanks! We'll give this a try.

Also, if it helps anyone else, here's how you can get current limits:

cbstats localhost:11210 all | grep  -e max_data_size -e mem_high_wat -e mem_low_wat

For more information about how the watermarks are used, read here:
http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-architectur...

As the system is loaded with data, eventually the mem_low_wat is passed. At this time, no action is taken. This is the "goal" the system will move toward when migrating items to disk. As data continues to load, it will evenutally reach mem_high_wat. At this point a background job is scheduled to ensure items are migrated to disk and the memory is then available for other Couchbase Server items. This job will run until measured memory reaches mem_low_wat. If the rate of incoming items is faster than the migration of items to disk, the system may return errors indicating there is not enough space. This will continue until there is available memory.

To dynamically set them based on percentages, we did:

# Set high/low water marks
mem_high_wat=$(ruby -e 'puts (%x{cbstats localhost:11210 all|grep max_data_size}.split(/[: ]+/)[2].to_i * 0.95).to_i')
mem_low_wat=$(ruby -e 'puts (%x{cbstats localhost:11210 all|grep max_data_size}.split(/[: ]+/)[2].to_i * 0.90).to_i')
 
cbflushctl localhost:11210 set mem_high_wat $mem_high_wat 
cbflushctl localhost:11210 set mem_low_wat $mem_low_wat