Send data to Scope/Collection with NiFi

Other
1

Hi!

I am trying to send data to a Scope/collection with the NiFi processor PutCouchbaseKey . But always give me the error "Could not open Bucket.

image
image1116Ă—616 31.6 KB

There is no option for scopes & collection
Is possible send data to a Collection or only to a buckets with NiFi?

Thanks!

2

Hi David. Welcome to the Couchbase Forum!

The current version of NiFi’s PutCouchbaseKey processor uses Couchbase SDK 2, which only supports a bucket’s default scope and collection. The processor would need to upgrade to Couchbase SDK 3 before it could support writing to a different scope/collection.

I will try to find out who owns PutCouchbaseKey (I suspect it’s the NiFi team), and see if we can help them upgrade to Couchbase SDK 3.

I do not know why you’re seeing “Could not open Bucket”. I wonder if there’s a log file somewhere with more details?

Thanks,
David N.

3

Hi David,

I am new user on Couchbase and Nifi, actually I am facing a same problem here, I have more then a hundred tables from the source, as the target, I don’t really wants to prepare more then hundred bucket on Couchbase. So I would like to asking a few questions.

  1. Any workaround can do it with moving data to scopes/collection instead of to bucket?
  2. Any roadmap of the release date for the next version of NIFI can be done with Couchbase SDK 3 supported?
  3. Or if writing to the bucket is the only solution, any performance impact to access those data across the bucket?

Many thanks!

Best regards,
Tim

4

Hi Tim,

As far as I know, the situation hasn’t changed in the last year.

  1. Any workaround can do it with moving data to scopes/collection instead of to bucket?

Not that I know of. As far as I can tell, the NiFi CouchbasePut operation still does not support collections.

  1. Any roadmap of the release date for the next version of NIFI can be done with Couchbase SDK 3 supported?

That might be a good question for the NiFi developers. See Apache NiFi Mailing Lists (one of the last links on that page is for their Slack workspace, which is quite lively).

  1. Or if writing to the bucket is the only solution, any performance impact to access those data across the bucket?

Are you asking about the performance impact of having upwards of 100 buckets? I’m not sure, but I don’t think it’s something we typically recommend.

I can’t promise anything, but I’ll make sure our Product Manager sees this post.

Thanks,
David

5

I’m not familiar with the capabilities of NiFi, so please forgive me if I’m about to spout nonsense. I wonder if instead of writing to different collections, you could somehow encode the ultimate collection name in the document key. For example, use some NiFi processing magic to change the document key “foo” into “myScope.myCollection.foo”… and then use NiFi’s CouchbasePut to write all the documents to the same collection (the target bucket’s _default collection).

Then on the Couchbase side, perhaps you could use the Eventing service (paging @jon.strabala) to scan all the incoming documents, chop off the collection name prefix, and copy the documents to the desired collections.

Thanks,
David

6

As David says Eventing can do this (not just by chopping up a key but it can determine the destination bucket via any data within the document itself).

Although an update to the COuchbase SDK3 in the NiFi processor you are using to natively support collections does seem to be the best long term route.

7

Thanks David, Jon and all the team from Couchbase, let me providing some update here.

Apache Nifi just release the first milestone version of 2.0.0 on 25, Nov 2023
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1

Unfortunately, a Couchbase SDK v2.7.3 still be used which not supporting scope and collection.

So i did following the suggestion by David, studying and running some test by Eventing service to scan all the incoming documents, chop off the document id by predefined syntax, and copy to a target scope and collection.

It can be an intermediate solution, for long term, hope that the Couchbase SDK will be upgraded on following release of Nifi.

1 Like