Distributing a bucket over multiple servers, and creating a view on it.
I have created my couchbase cluster over a few servers and because I didn't want to create different views over more than one bucket so I tried to have all my data in one bucket, but the problem I'm facing is that while inserting the data and after exceeding 50% of the available space on all servers couchbase started to alert me about that :
"Metadata overhead warning. over 52% of RAM allocated to bucket "default" on node ".... " is taken up by keys and meta data".
Now I know that I could solve the problem by creating more than one bucket but then querying data from an application would be a problem as I need to query all the views created on all buckets and gather the results instead of querying one view.
Is there a way of either having all my data in one bucket? or if not, is there a way to create a view over multiple buckets so couchbase can deal with querying them all automatically? if not , what could be the optimal way to deal with this.
Again, my aim is to reduce the amount to processing I'm doing inside my application.
Let me answer your question in 2 steps:
When you have "issue with memory", adding new bucket will not help you, you just need to add more RAM to your bucket, and for this you have 2 ways:
- add new nodes to your cluster
- add RAM to your bucket
- or both ;)
Since you are ready to add new bucket, this means you have more memory available for you cluster, so just add more RAM to your bucket and this will be it. This message is just to tell you that you start to have lot of metadata in RAM compare to the volume of your working set. As you know Couchbase keeps all the keys and meta data in RAM to make it very fast to find information in the database. By default this warning is raised when you reach 50% you can change it, but usually it is a very useful warning to look at the way your application.
The less bucket you have in your cluster the better it will be, we usually say do not create more than 10 buckets in one cluster. Keep in mind that most of the operations such as indexing, replication, compaction... are done per bucket so the more bucket you have, more resource you will consume of the server(s) see:
You are asking if it is possible to use a view (and query it) over 2 buckets, the answer is simple and it is no. It is not possible to write a view for 2 buckets and query it.
Most of the time people are creating buckets when:
- they have security constraints, just to be sure that data are separated
- they have really different type of use cases: for example a part of the application is using documents, with views, and another part of the application is using just simple key value with TTL,.. and no views.
The views will be also automatically distributed on all the nodes of the cluster since the indexing process is done where the document is located.
You can find more information about Couchbase architecture in this white paper:
Let me know if you have questions