{"id":3517,"date":"2024-04-17T11:51:58","date_gmt":"2024-04-17T18:51:58","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/"},"modified":"2024-04-17T11:51:58","modified_gmt":"2024-04-17T18:51:58","slug":"streaming-data-with-amazon-msk-and-couchbase-capella","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/","title":{"rendered":"Streaming Data with Amazon MSK and Couchbase Capella"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><span>We see that a significant number of customers take advantage of Couchbase integration with <\/span><a href=\"https:\/\/kafka.apache.org\/\"><span>Apache Kafka<\/span><\/a><span>, by using Couchbase Kafka connector plugin that provides you capability to reliably stream data to and from Apache Kafka at scale.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications. However, you need Apache Kafka infrastructure management expertise to architect, operate, and manage on your own. Amazon Managed Streaming for Apache Kafka (<\/span><a href=\"https:\/\/aws.amazon.com\/msk\/\"><span>Amazon MSK<\/span><\/a><span>) is a fully managed, highly available service that makes it easy for you to build and run applications that use Apache Kafka.\u00a0<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Amazon MSK supports integration with Couchbase with the Amazon MSK Connect feature and Couchbase Kafka connector plugin. Using this feature, you can easily deploy the Couchbase connector and scale to adjust for changes in load.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201c<\/span><i><span>sink<\/span><\/i><span>\u201d and \u201c<\/span><i><span>source<\/span><\/i><span>\u201d. We will be using Couchbase Capella to get us started in minutes.<\/span><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Step 1: Couchbase Capella cluster<\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span>Get Started with <\/span><a href=\"https:\/\/cloud.couchbase.com\/sign-up\"><span>Couchbase Capella free trial<\/span><\/a><span>.<\/span><\/li>\n\n\n<li><span>Select your preferred AWS region and get started with the Couchbase Capella cluster in minutes.<\/span><\/li>\n\n\n<li><span>Next, <\/span><a href=\"https:\/\/docs.couchbase.com\/cloud\/clusters\/manage-database-users.html\"><span>configure Database Credentials<\/span><\/a><span>.<\/span><\/li>\n\n\n<li><span>Setup private networking using <\/span><a href=\"https:\/\/docs.couchbase.com\/cloud\/clouds\/private-network.html\"><span>VPC Peering<\/span><\/a><span> or <\/span><a href=\"https:\/\/docs.couchbase.com\/cloud\/security\/add-aws-private-link.html\"><span>AWS PrivateLink<\/span><\/a><span> for network connectivity with your AWS account. You can allow access from anywhere but it\u2019s not recommended.\u00a0\u00a0<\/span><\/li>\n\n\n<li><span>Next, access the Data Tools section to create a new <\/span><strong><em>demo<\/em><\/strong><span> Bucket on the cluster.<\/span><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Step 2: Amazon MSK cluster and Apache Kafka client<\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span>It\u2019s recommended that you have your Amazon MSK cluster and Amazon MSK Connect connectors in the private subnet of your VPC. In case you are not using VPC Peering or AWS PrivateLink to connect to Couchbase Capella cluster you would need the NAT gateway to initiate connection from the private subnets.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>To get started easily we can use <\/span><a href=\"https:\/\/docs.aws.amazon.com\/solutions\/latest\/streaming-data-solution-for-amazon-msk\/deploy-template1.html#step1\"><span>CloudFormation template for Streaming Data Solutions for MSK<\/span><\/a><span> which deploys an Amazon MSK cluster and Apache Kafka client on Amazon EC2 instance.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-15599\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image12-1.png\" alt=\"\" width=\"144\" height=\"27\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image10-4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15600\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image10-4.png\" alt=\"\" width=\"490\" height=\"274\"><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Let\u2019s first create the sample sink and source topics on the MSK cluster:<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image6.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15607\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image6-1024x260-1.png\" alt=\"\" width=\"668\" height=\"170\"><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Next, we will install few packages of the <\/span><b>KafkaClient<\/b><span> instance so that we can run a sample code that will publish a few messages on the <\/span><b>couchbase-sink-example <\/b><span>topic<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image14.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15608\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image14-1024x172-1.png\" alt=\"\" width=\"745\" height=\"125\"><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>We will follow the <\/span><a href=\"https:\/\/docs.couchbase.com\/kafka-connector\/current\/quickstart.html#sink\"><span>QuickStart guide for Couchbase Sink Connector<\/span><\/a><span> to publish messages to the MSK cluster.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image11.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15609\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image11-1024x87-1.png\" alt=\"\" width=\"793\" height=\"67\"><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Open the file src\/main\/java\/com\/couchbase\/connect\/kafka\/example\/JsonProducerExample.java and update Kafka Cluster Bootstrap connection string.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>Now you can build the maven project:<\/span><\/p>\n\n\n<p>[crayon nums=&#8221;false&#8221; lang=&#8221;default&#8221; decode=&#8221;true&#8221;]mvn compile exec:java[\/crayon]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><span>The sample producer will send some messages to the kafka topic <\/span><b>couchbase-sink-example<\/b><span> and then terminate.<\/span><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Step 3: Configure MSK Connect plugin<\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image15-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15601\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image15-2.png\" alt=\"\" width=\"556\" height=\"630\"><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span>Download the <\/span><a href=\"https:\/\/docs.couchbase.com\/kafka-connector\/current\/release-notes.html\"><span>Couchbase Kafka Connect plugin<\/span><\/a><span> ZIP.<\/span><\/li>\n\n\n<li><span>Upload the ZIP file to an S3 bucket to which you have access.<\/span><\/li>\n\n\n<li><span>Open the <\/span><a href=\"https:\/\/console.aws.amazon.com\/msk\/home?region=us-east-2#\/home\/\"><span>Amazon MSK console<\/span><\/a><span>. In the left pane expand <\/span><b>MSK Connect<\/b><span>, then choose <\/span><b>Custom plugins.<\/b><\/li>\n\n\n<li><span>Choose <\/span><b>Create custom plugin<\/b><\/li>\n\n\n<li><span>Choose\u00a0<\/span><b>Browse S3<\/b><span>. In the list of buckets find the bucket where you uploaded the ZIP file, and then in the list of objects select the ZIP file.<\/span><\/li>\n\n\n<li><span>Enter <\/span><b>couchbase-kafka-connect<\/b><span> as custom plugin name, then choose Create custom plugin.<\/span><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Step 4: Create MSK Connector for Sink<\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image1-2.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15603\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-1024x350-1.png\" alt=\"\" width=\"710\" height=\"243\"><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span>Using the custom plugin, we can now create a sink connector. <\/span><span>Open the <\/span><a href=\"https:\/\/console.aws.amazon.com\/msk\/home?region=us-east-2#\/home\/\"><span>Amazon MSK console<\/span><\/a><span>. In the left pane, under <\/span><b>MSK Connect<\/b><span>, choose <\/span><b>Connectors<\/b><span>. Choose <\/span><b>Create connector<\/b><span>.<\/span><\/li>\n\n\n<li><span>Choose the custom plugin <\/span><b>couchbase-kafka-connect<\/b><i><span> and choose Next.<\/span><\/i><\/li>\n\n\n<li><span>Enter<\/span> <b>couchbase-sink-example<\/b><span> as <em>Connector<\/em> name.<\/span><\/li>\n\n\n<li><span>Select the MSK Cluster created in Step 2.<\/span><\/li>\n\n\n<li><span>Enter the connector configuration shown below:<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image9.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15610\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image9-1024x460-1.png\" alt=\"\" width=\"653\" height=\"293\"><\/a><br>\n<\/span><\/li>\n\n\n<li><span>Select <em>Provisioned<\/em> or <em>Autoscaled<\/em> capacity based on your requirement.<\/span><\/li>\n\n\n<li><span>Create and select an IAM role with the <\/span><a href=\"https:\/\/docs.aws.amazon.com\/msk\/latest\/developerguide\/msk-connect-service-execution-role.html\"><span>required trust policy<\/span><\/a><span> for MSK connect.<\/span><\/li>\n\n\n<li><span>Once the <\/span><b>couchbase-sink-example<\/b><span> connector starts you will see messages from the Kafka topic added to the Couchbase bucket as new documents.<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image7.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15604\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image7-1024x213-1.png\" alt=\"\" width=\"669\" height=\"139\"><\/a><br>\n<\/span><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Step 5: Create MSK Connector for Source<\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image8.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15602\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image8-1024x345-1.png\" alt=\"\" width=\"734\" height=\"247\"><\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><span>Using the custom plugin, we can now create a source connector. <\/span><span>Open the <\/span><a href=\"https:\/\/console.aws.amazon.com\/msk\/home?region=us-east-2#\/home\/\"><span>Amazon MSK console<\/span><\/a><span>. In the left pane, under <\/span><b>MSK Connect<\/b><span>, choose <\/span><b>Connectors<\/b><span>. Choose <\/span><b>Create connector<\/b><span>.<\/span><\/li>\n\n\n<li><span>Choose the custom plugin <\/span><b>couchbase-kafka-connect<\/b><span> and choose<\/span><i><span> Next.<\/span><\/i><\/li>\n\n\n<li><i><span>Enter <\/span><\/i><b>couchbase-source-example<\/b><span> as <em>Connector<\/em> name<\/span><\/li>\n\n\n<li><span>Select the MSK Cluster created in Step 2.<\/span><\/li>\n\n\n<li><span>Enter the connector configuration shown below:<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image2-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15611\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image2-1-1024x577-1.png\" alt=\"\" width=\"559\" height=\"315\"><\/a><\/span><\/li>\n\n\n<li><span>Select Provisioned or Autoscaled capacity based on your requirement.<\/span><\/li>\n\n\n<li><span>Create and select an IAM role with the <\/span><a href=\"https:\/\/docs.aws.amazon.com\/msk\/latest\/developerguide\/msk-connect-service-execution-role.html\"><span>required trust policy<\/span><\/a><span> for MSK connect.<\/span><\/li>\n\n\n<li><span>Once the <\/span><b>couchbase-source-example<\/b><span> connector starts it will be listening to Couchbase Database Change Protocol (DCP).<\/span><\/li>\n\n\n<li><span>Let\u2019s start a console consumer on the <\/span><b>KafkaClient<\/b><span> instance to listen to the Kafka topic <\/span><b>couchbase-source-example <\/b><span>where the update messages will be published. <\/span><span>For this I will again connect to the KafkaClient instance using the Session Manager option and run the following code:<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image13.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15612\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image13-1024x144-1.png\" alt=\"\" width=\"603\" height=\"85\"><\/a><br>\n<\/span><\/li>\n\n\n<li><span>Now when we run UPDATE queries on the <\/span><b>travel-sample<\/b><span> bucket on Couchbase Capella using the Data Tools \u2013 Query option:<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image5-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-15613\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image5-1-1024x87-1.png\" alt=\"\" width=\"722\" height=\"61\"><\/a><br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image4-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-15605\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image4-1-1024x270-1.png\" alt=\"\" width=\"900\" height=\"237\"><\/a><br>\n<\/span><\/li>\n\n\n<li><span>We can see the updated document is received by the Kafka consumer running on the Kafka Client instance:<br>\n<a href=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2024\/04\/image3-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-15606\" src=\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/image3-1-1024x149-1.png\" alt=\"\" width=\"900\" height=\"131\"><\/a><br>\n<\/span><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span>Conclusion<\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><span>This post illustrates how you can use the Couchbase <em>source connector<\/em> for publishing document change notifications from Couchbase Capella to a Kafka topic, as well as a <em>sink connector<\/em>\u00a0that subscribes to one or more Kafka topics and writes the messages to Couchbase Capella.<\/span><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><br><br><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We see that a significant number of customers take advantage of Couchbase integration with Apache Kafka, by using Couchbase Kafka connector plugin that provides you capability to reliably stream data to and from Apache Kafka at scale. Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications. However, you need Apache [&hellip;]<\/p>\n","protected":false},"author":84206,"featured_media":3514,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"_acf":"","footnotes":""},"categories":[178,301],"tags":[419,616],"ppma_author":[617],"class_list":["post-3517","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-connectors","category-cloud","tag-amazon-web-services-aws","tag-apache-kafka"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.6 (Yoast SEO v27.6) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Streaming Data with Amazon MSK and Couchbase Capella - The Couchbase Blog<\/title>\n<meta name=\"description\" content=\"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Streaming Data with Amazon MSK and Couchbase Capella\" \/>\n<meta property=\"og:description\" content=\"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-04-17T18:51:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1292\" \/>\n\t<meta property=\"og:image:height\" content=\"442\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Saurabh Shabhag, Partner Solutions Architect, AWS\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Saurabh Shabhag, Partner Solutions Architect, AWS\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/\"},\"author\":{\"name\":\"Saurabh Shabhag, Partner Solutions Architect, AWS\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#\\\/schema\\\/person\\\/a8aa6564b05e34cfb9e44458b46886c6\"},\"headline\":\"Streaming Data with Amazon MSK and Couchbase Capella\",\"datePublished\":\"2024-04-17T18:51:58+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/\"},\"wordCount\":859,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/05\\\/image1-2-7.png\",\"keywords\":[\"Amazon Web Services (AWS)\",\"Apache Kafka\"],\"articleSection\":[\"Connectors\",\"Couchbase Capella\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/\",\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/\",\"name\":\"Streaming Data with Amazon MSK and Couchbase Capella - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/05\\\/image1-2-7.png\",\"datePublished\":\"2024-04-17T18:51:58+00:00\",\"description\":\"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/05\\\/image1-2-7.png\",\"contentUrl\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/05\\\/image1-2-7.png\",\"width\":1292,\"height\":442},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/streaming-data-with-amazon-msk-and-couchbase-capella\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Streaming Data with Amazon MSK and Couchbase Capella\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/06\\\/logo.svg\",\"contentUrl\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/5\\\/2026\\\/06\\\/logo.svg\",\"width\":\"1024\",\"height\":\"1024\",\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/#\\\/schema\\\/person\\\/a8aa6564b05e34cfb9e44458b46886c6\",\"name\":\"Saurabh Shabhag, Partner Solutions Architect, AWS\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g79cc939ef54f5763ebf78afdb9e30cb0\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g\",\"caption\":\"Saurabh Shabhag, Partner Solutions Architect, AWS\"},\"url\":\"https:\\\/\\\/www.couchbase.com\\\/blog\\\/author\\\/awssaurabh\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Streaming Data with Amazon MSK and Couchbase Capella - The Couchbase Blog","description":"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/","og_locale":"en_US","og_type":"article","og_title":"Streaming Data with Amazon MSK and Couchbase Capella","og_description":"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.","og_url":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/","og_site_name":"The Couchbase Blog","article_published_time":"2024-04-17T18:51:58+00:00","og_image":[{"width":1292,"height":442,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png","type":"image\/png"}],"author":"Saurabh Shabhag, Partner Solutions Architect, AWS","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Saurabh Shabhag, Partner Solutions Architect, AWS","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/"},"author":{"name":"Saurabh Shabhag, Partner Solutions Architect, AWS","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/a8aa6564b05e34cfb9e44458b46886c6"},"headline":"Streaming Data with Amazon MSK and Couchbase Capella","datePublished":"2024-04-17T18:51:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/"},"wordCount":859,"commentCount":1,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png","keywords":["Amazon Web Services (AWS)","Apache Kafka"],"articleSection":["Connectors","Couchbase Capella"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/","url":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/","name":"Streaming Data with Amazon MSK and Couchbase Capella - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png","datePublished":"2024-04-17T18:51:58+00:00","description":"In this blog post, we will go through Amazon MSK cluster setup and use Couchbase Kafka Connector as \u201csink\u201d and \u201csource\u201d. We will be using Couchbase Capella to get us started in minutes.","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image1-2-7.png","width":1292,"height":442},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/streaming-data-with-amazon-msk-and-couchbase-capella\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Streaming Data with Amazon MSK and Couchbase Capella"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/06\/logo.svg","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/06\/logo.svg","width":"1024","height":"1024","caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/a8aa6564b05e34cfb9e44458b46886c6","name":"Saurabh Shabhag, Partner Solutions Architect, AWS","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g79cc939ef54f5763ebf78afdb9e30cb0","url":"https:\/\/secure.gravatar.com\/avatar\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4839f566158817b7f9c51e4cc9fc8ac2675a6de9f2b0ea50d3be032daf46f345?s=96&d=mm&r=g","caption":"Saurabh Shabhag, Partner Solutions Architect, AWS"},"url":"https:\/\/www.couchbase.com\/blog\/author\/awssaurabh\/"}]}},"acf":[],"authors":[{"term_id":617,"user_id":84206,"is_guest":0,"slug":"awssaurabh","display_name":"Saurabh Shabhag, Partner Solutions Architect, AWS","avatar_url":{"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image_2023-02-28_133158165-5.png","url2x":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/05\/image_2023-02-28_133158165-5.png"},"author_category":"","first_name":"Saurabh","last_name":"Shabhag","user_url":"","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/3517","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/84206"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=3517"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/3517\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/3514"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=3517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=3517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=3517"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=3517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}