The current version of NiFi’s PutCouchbaseKey processor uses Couchbase SDK 2, which only supports a bucket’s default scope and collection. The processor would need to upgrade to Couchbase SDK 3 before it could support writing to a different scope/collection.
I will try to find out who owns PutCouchbaseKey (I suspect it’s the NiFi team), and see if we can help them upgrade to Couchbase SDK 3.
I do not know why you’re seeing “Could not open Bucket”. I wonder if there’s a log file somewhere with more details?
I am new user on Couchbase and Nifi, actually I am facing a same problem here, I have more then a hundred tables from the source, as the target, I don’t really wants to prepare more then hundred bucket on Couchbase. So I would like to asking a few questions.
Any workaround can do it with moving data to scopes/collection instead of to bucket?
Any roadmap of the release date for the next version of NIFI can be done with Couchbase SDK 3 supported?
Or if writing to the bucket is the only solution, any performance impact to access those data across the bucket?
I’m not familiar with the capabilities of NiFi, so please forgive me if I’m about to spout nonsense. I wonder if instead of writing to different collections, you could somehow encode the ultimate collection name in the document key. For example, use some NiFi processing magic to change the document key “foo” into “myScope.myCollection.foo”… and then use NiFi’s CouchbasePut to write all the documents to the same collection (the target bucket’s _default collection).
Then on the Couchbase side, perhaps you could use the Eventing service (paging @jon.strabala) to scan all the incoming documents, chop off the collection name prefix, and copy the documents to the desired collections.
Unfortunately, a Couchbase SDK v2.7.3 still be used which not supporting scope and collection.
So i did following the suggestion by David, studying and running some test by Eventing service to scan all the incoming documents, chop off the document id by predefined syntax, and copy to a target scope and collection.