Working through my automated process of copying Oracle tables to buckets, I’ve now discovered that there’s a limit of 10 buckets in a cluster. How can I override this limit?
The sizes of my tables that I’m copying vary widely. Most of them are quite small. Some of them are quite big (relatively). I know that I don’t have to have a 1-1 relation from tables to buckets, but doing so makes my transfer process very convenient. It will be more complicated to combine multiple tables into a single bucket.
Someone on #couchbase pointed me to https://groups.google.com/forum/#!topic/couchbase/wbMJgGlvA64 , which might have worked. My script takes quite a while to run, so I’ll have to wait to see whether this worked.
I’m well aware of the fact that there are good reasons to limit this. My use case is a little odd.
@davidmichaelkarr,
I think your not going to get the performance your looking if you have a 1 to 1 ratio of Tables to Buckets
Here is a Youtube video of a Couchbase customer, Brant Burnett - Software Developer at CenterEdge, talking about RDBMS to Couchbase migrations they did.
Here is a Getting Started pdf: https://www.couchbase.com/binaries/content/assets/website/docs/whitepapers/moving-from-relational-to-nosql-how-to-get-started.pdf
I’m sure you’re right, if this was a conventional Couchbase application. In this case, Couchbase is only being used to bulk load into a set of maps in memory once a night. The process will read every document in one bucket, then go on to the next bucket. That’s all it does. There aren’t even any relations between documents in different buckets. Yes, it’s an embarrassing use case for Couchbase, but the platform architecture I have to work with only has NoSQL persistence, so I have to jam my use case into Couchbase.
@davidmichaelkarr,
Can you name space your key’s and put them in one bucket or a few.
EXAMPLE
table1:(key1) … tabel2:(key1)
That way you don’t trample over each other key if they are not 100% unique across tables.
You can use indexes with WHERE predicate to partition data maybe by type of customer / user ID
@davidmichaelkarr
In RDBMS, it’s normal to have hundreds or thousands of tables in a database.
Where there are ways to increase the number of buckets in Couchbase, that’s not the recommended approach.
As @househippo suggested, you should design your document keys and queries so you can use a single bucket. We have customers with billions of documents in a single bucket.
I’ve written a bit of commentary on this in a blog:
https://blog.couchbase.com/couchbase-oracle-developers-part-3-database-objects/