Minimum Bucket Size
Hi,
From my understanding, the current minimum bucket size is 64Mb. This setting is per node in a cluster and the allocated bucket memory becomes unusable to other buckets. I can't create a set of buckets whose summed memory allocation exceeds the quota per node.
Let's say I have 2000 tenants/buckets, thats ~124GB of memory needed per node given the min bucket size. That doesn't feel very scalable to me.
I hope I am misunderstanding how these buckets can be configured because these restrictions place some architectural hardships on us.
Any thoughts/comments would be greatly appreciated.
Thanks!
HI Frank,
at this point if you want 1000's of small buckets we recommend you create multiple clusters.
We found that depending on the use case people have very different requirements. Most typically we found that users want their buckets to grow in size as they add more nodes to a cluster.
Fixing the size of a bucket, leads to a smaller and smaller portion of each bucket residing on each node, and there are certain data structures that are needed, which means that the amount of RAM needed per node can't shrink towards zero. So having a minimum amount of RAM needed per node is required.
We will provide the ability to restrict the number of nodes used by a bucket to a subset of nodes in the cluster at some point, but that raises all kind of other problems that need to find solutions (especially on what to do when a specific node fails, as none of the remaining nodes might have enough spare RAM for hat subset of buckets that only resided on that node....)
It would be interesting to hear more about your use-case, it sounds like you might want to offer membase as a service, where you expect lots of users with very small buckets? What bucket size do you expect your users will want? How many nodes did you envisage to be in your cluster?
As stated in the beginning of the post, right now I'd suggest you create small clusters and once a cluster has filled up with users, add a second cluster.
Cheers,
Frank
I too have the same use-case. The issue for me is that many clients may be allocated a bucket, but never or rarely use it. I would like to either:
- Not specify the amount of memory allocated to a bucket
- Not have membase restrict the allocation of buckets to only 100% of the cluster quota
Thanks!
Thanks!
As we continue to build out the multi-tenancy features of Membase these requests will be taken into account.
Perry
Actually, for Membase bucket types, the minimum is 256mb...we raised that after realizing that 64mb was not enough. For Memcached-type buckets the limit is still 64mb.
Your concern does seem valid however. We are actually working on the ability to 'pin' buckets to certain nodes so you don't have to spread each customer out across the whole cluster so you would just need 124GB total.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!