Hi ! I am looking to connect to Pub/Sub as a topic.
I have used the Connectors on GCP, but it is very difficult to set up. Could you please, tell me what the authentication is: there is a DNS server and a SSLServerCert to enter, and I don’t know what they are refering to when one is using Capella. On the other hand, I am not sure this is the right way to do so and I would like to know how to set up a connection directly.
Thanks a lot for your answer.
Best regards,
Sarah
The “Connect” tab will provide the information you need to connect. https://cloud.couchbase.com/sign-in
The default capella server cert is built into the SDK - you don’t need to specify it.
Here’s the doc on connecting to Capella - each sample has a Capella tab.
Would it be possible to connect to pubsub ? i would have to create a function in cloud function that connects to capella server and gets the new data in a certain time frame then sends message to topic ?
How do I get the streaming data (for instance new records directly arrived at Capella) - What would be the purpose of this?
Thanks a lot for your answer.
Sarah
You want a couchbase connector - Connector Guides | Couchbase Docs
For pub/sub there does not seem to be one.
What would you suggest?
It sounds like you’re after a message oriented middleware (e.g. Websphere MQ, Rabbit MQ, etc.). Couchbase could be used as the back-end datastore for a messaging engine, but isn’t one itself.
HTH.
That might be it - but I am really looking to stream data to couchbase as a datalake and then each of these messages or written records would be published to a topic with Pub/Sub (hence the necessity of a connection) - and it will be sent to subscribers for instance bq or anything like this. But I was also wondering what could be the purpose and advantage of such an architecture
It would not be temporary storage of messages, but more like an additional transaction to feed an ETL pipeline for instance and keep the basic records in Couchbase
“Publish to a topic” is a messaging concept, ergo the pointer to MQ software. Unless I’ve just misunderstood and you’re not meaning a managed published topic that clients will connect to and “subscribe” to receive all items sent to the topic ?
If all you’re after the equivalent of an SQL trigger (perform some action for each inserted document, for example), then Eventing functions will do what you need. e.g. Push (select) information from newly inserted documents to an external ETL feed. A lot of the necessary details for such a scenario would depend on exactly how the external pipeline is fed.
https://docs.couchbase.com/server/current/eventing/eventing-examples.html
You could combine the two too - an Eventing function to write to MQ published topics, etc.
MQ pub-sub has the advantage (among others) that the MQ client would have a routine that is waiting for notifications from the MQ server, when connected. This saves it from polling for events and thus can ease some client coding scenarios. The MQ middleware also provides management for clients etc. and a pub-sub client would typically receive all unseen - i.e. including those that occurred when disconnected - events (when connected). This could be a disadvantage too, as it could drive a flurry of work when connecting.
A non-MQ client could say establish an HTTP server and listen for requests as a means of receiving notifications, if it doesn’t care for events that occur when it isn’t running. Client management would not be taken care of in this case, but Eventing functions could still be written to notify such a client.
HTH.
Imagine as a source that you would be tracking modifications in records in the address field in a client database. Everytime there is a change the record is sent in a publishing system to send to an email notification system. As would Kafka do for this situation with receiving the info of an order for instance, how would I do this passing through Capella ? - I am not even sure this architecture could be useful the best architecture, but I would like to do this with Pub/Sub and not Kafka with Capella and not the Couchbase server. Is this possible ?
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.