RAM Sizing formulas are incorrect?
Hello!
I have a problem with calculation RAM quota.
I use formulas on http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractic...
For example I need to create 1 bucket on node...
documents_num = 200000
ID_size = 7
value_size =65
number_of_replicas = 0
working_set_percentage = 100% (all data in RAM)
per_node_ram_quota = 1000Mb
Meta data per document = 120
headroom = 25%
High Water Mark = 70%
So, using formulas I've got 66 Mb Claster RAM quota required.
After that I create bucket with 100 Mb RAM quota (100 > 66, so I thought It would enough). But then I created a bucket
and began to store values ... memory usage parameter rises faster then "user data memory". After that it exceeded "High water mark" parameter, and store key function returned error: "Temporary error. Please try later".
But If I set 200 Mb ram quota for bucket - all is good and all keys have been stored.
So, how I should compute RAM quota? Or maybe it is a bug with memory usage parameter (Why It uses additional memory that needed)?
Hello Alex.
Thanks for your reply.
I looked at the overall memory usage when I was at the 200 Mb bucket and it was 120 Mb... So it exceed your calculated 80-90mb.
But it went down to 105 Mb after about 5 minutes left.
And one more question. Why doesn't metadata parameter change its value? It's always display 25.5 Mb (But according to my calculations it must be 31 Mb)?
in 1.8 - we generally pad the forumla for metadata(metadata size is not actually 150bytes) to account for memory accounting issues as well as memory used by the data structures we use for replicating to other nodes or draining to disk so its not a 100% exact in 1.8 and its not the best way to do it but its how we generally size for a 1.8x cluster
Its much closer to exact on the 2.0 release.
Hi there.
The docs are a bit off here. For 1.x - we generally use 150 bytes for the calculation. For 2.0 we use 75 bytes and this will be going even lower in later releases.
So, changing the math a bit(high watermark is at 75%), I get about 74mb for the cluster ram quota. 31mb of that being key name + metadata, 44mb for the values.
This gets you just under the high watermark and without a replica, nothing should be ejected. However, 1.8.x is less accurate when it comes to memory accounting than the 2.0 release.
In 1.8, with 74mb of total data, it may actually use 80mb-90mb because of how we allocate and deallocate memory. In 2.0, you might be at 74.5-75mb so its much better from that standpoint.
One question for you - is did you look at the overall memory usage when you were at the 200mb bucket? This will give you a better sense of it and of course, use the higher numbers for watermark/metadata in your calculation.