Capella App Services: Enhancing Observability with Real-Time Log Streaming Support

In an earlier post, we discussed the fundamentals of the Log Streaming feature on Capella App Services. App Services logs can be streamed in real-time to third-party observability platforms such as Datadog or collectors, hosted in customer premises.

Log Streaming allows you to gain instant insights into application behavior, enabling rapid issue detection and resolution. This enhancement is key to improving your application’s reliability, availability, and security.

In this blog post, I’ll guide you on how to easily stream logs from App Services to Datadog’s Log Management solution to enable log collection, parsing and normalization, visualization and proactive logs analysis. 

This blog post assumes you are familiar with the capabilities of real-time Log Streaming in App Services. If you need a refresher, catch up with the first part of this blog on Capella App Services: Enhancing Observability with Real-Time Log Streaming Support. 

Prerequisites

Before diving into the tutorial, ensure that you have the following prerequisites in place:

Real-time log streaming from Capella App Services to Datadog

Datadog is a leading cloud-based monitoring and analytics platform that aids in observing, troubleshooting, and optimizing their applications’ performance. Within the context of this tutorial, we’ll focus on Datadog’s capabilities of logs monitoring. 

Logs are essential data streams that capture event-specific details about software applications, systems, and platforms. By monitoring logs in real-time, teams can gain immediate insights into system behaviors, spot anomalies, troubleshoot issues, and ensure optimal system health. 

Streaming real-time logs offers visibility into the operations of your applications and is simple to set up. By connecting Couchbase’s Capella App Services to Datadog and utilizing App Services Log Streaming, you can have a centralized place to store your logs, that supports advanced processing and normalization, visualization and dashboards as well as alerting. 

This integration not only simplifies the setup process but also enhances the ability to monitor, analyze, and respond to the logs your applications generate, ensuring proactive insights extraction and issue resolution.

How to set up log streaming to Datadog

Step 1: Setup in Datadog

    • Log in to your Datadog account.
    • Navigate to Logs.
      • Look for the ‘Logs’ section in the Datadog dashboard. 
    • Select Fluent Bit as your log source.
      • This will show you the details you’ll need to set up streaming from Capella App Services. Specifically, you will see:
        • Host Address: This is the endpoint where the logs will be sent.
        • API Key: An authorization token to authenticate your log streams.

Step 2: Setup in Capella App Services

    • Log in to your Couchbase Capella dashboard.
    • Navigate to your App Service’s Settings page.
    • Find and select the Log Streaming option.
      • Here, you’ll configure where and how your logs will be sent.
    • Configure Datadog as your Log Collection Provider:
      • For Collection URL, enter the Host Address you obtained from Datadog.
      • In Access Credentials, input the Datadog API Key you noted earlier.

    • Enable Log Streaming:
      • Click on the option to enable log streaming. This might take a moment as it sets up the collector.
      • Monitor the status. Once fully set up, you should see the status transition to “Enabled”, indicating that logs are now streaming from Capella App Services to Datadog.

    • Once enabled, App Services Log Streaming can be then disabled or paused. 
      • Pausing temporarily stops streaming logs to the log collector until streaming is resumed.
      • Disabling stops the streaming of logs and clears the configuration. 
      • Configuration can also be changed if any of the log collector’s target URLs or secrets change.
    • (Optional) Powerful App Endpoints level configurability
      • Variety of advanced Log Filters can be configured such as Auth, Import, Sync and many more to enhance the log types that are captured and streamed.
      • The Log Level (e.g. Info, Warning and Error) can also be altered for a given App Endpoint.

Step 3: View and Analyze Logs in Datadog

    • Go back to your Datadog dashboard.
    • Navigate to the ‘Live Tail’ section within ‘Logs’. 
      • This section gives you a real-time view of incoming logs. 
      • You should now see a real-time stream of logs coming in from Capella App Services.

    • (Recommended) Set up a Datadog Logs Pipeline:
      • This is beneficial if you want to translate the logs from App Services into a common format that suits your monitoring needs within Datadog. Follow Datadog’s documentation or guidelines on how to set this up based on your specific requirements.

Wrap Up

Congratulations! You’ve now successfully set up real-time log streaming from Couchbase Capella App Services to Datadog. You can now monitor, analyze, and act upon the insights you gain from your logs. Remember to keep an eye on your logs and set up any alerts or further integrations as needed to maximize your monitoring capabilities.

What’s Next

In this post we described the fundamentals of real-time Log Streaming in Capella App Services . In a future post, we will go through a step-by-step tutorial on how to easily stream App Services operational logs to Sumologic. Stay tuned!

If you have questions or feedback, please leave a comment below or email me. The Couchbase Forums or Discord are another good place to reach out with questions.

Catch up with the rest of the posts in this series on Capella App Services Log Streaming:

Author

Posted by Iveta Dulova, Senior Product Manager

All Posts

Leave a reply