I have a couchbase enterprise system with 8 buckets, I want to sync this 8 buckets data to Elastic search.
I went through the documentation for the couchbase - elastic search connector, but it can sync one bucket only with one connector app and need to spin more app for other buckets.
Is there any other way that I can use to sync the data to elastic?
I have gone through the sync gateway documentation, but not sure, whether it will work in this case.
Hi Ashish,
I think the officially supported path is to run one instance of the connector for each bucket. I appreciate that setting it up and administering it might be more complicated that you’d like.
It might be possible to rig something up using the Eventing Service and the Elasticsearch Indexing REST API. Is that something we’ve done before, @jon.strabala ?
Thanks,
David
Another way to simplify the deployment would be to upgrade to Couchbase 7 and store all of the documents in a single bucket, using separate Scopes and/or Collections instead of separate buckets. Of course, your existing applications would need to be updated as well, and perhaps that’s a longer-term project.
Although a bit heavy duty I think for lower data velocities, you could put an Eventing function on each of the eight buckets (or eight collections) with each tagging the data with it’s bucket/collection source and writing the data out to yet another staging bucket/collection where you then use the Couchbase Elasticsearch connector.
Best
Jon