In recent months the Kafka connector has seen several updates, adding features like log redaction in version 3.2.3, configurable durability requirements in 3.2.2, and a new SourceHandler extension point for custom processing in 3.2.0.

Now, with the release of Couchbase Server 5.5 right around the corner, we’ve updated the Kafka connector again to take advantage of some exciting new features available in the Enterprise Edition.

Add IPv6 to your bag of tricks (or, “Like two IPs in a quad”)

The industry-wide migration to IPv6 is in full swing. As Perry Krug wrote in March, Couchbase will join the party with support for IPv6 in the upcoming 5.5 release.

Version 3.2.3 of the Kafka connector could be made to work with IPv6 by setting a Java system property. This was inconvenient, so in version 3.3 we’ve enabled IPv6 support by default.

In most cases, the connector will continue to work with IPv4 without requiring any config tweaks. However, if you find yourself in an dual-stack environment where you need to force hostname resolution to use IPv4, you can get the desired behavior by setting the couchbase.forceIPv4 config property to true.

Data compression for enhanced resource utilization (or, “Just Squeeze Me”)

Chaitra Ramarao wrote a great introduction to the data compression features coming in Couchbase Server 5.5. Performance testing revealed that enabling compression can significantly improve throughput, so we’ve enabled compression by default in Kafka connector 3.3.

If you want to tune the compression settings, you remain in full control. The couchbase.compression config property lets you specify the compression mode to use. There are two modes to choose from (three if you count “disabled”). Let’s go over them in detail.

The new default mode is ENABLED. In this mode, you’re giving Couchbase Server permission to send data to the Kafka connector in compressed form. It might still send an uncompressed document if compressing it would require the server to do more work. However, most of the time the server will already have easy access to the compressed form, and the document will be sent compressed. We made this the default mode after our performance tests showed an overall improvement when compression is used at the cluster’s discretion.

The second compression mode is FORCED. In this mode, you’re telling Couchbase Server that network bandwidth is at a premium, and you want the server to make every effort to send documents in compressed form. As you might expect, this mode requires the server to do more work, so you’ll be using some additional CPU at the cluster side in exchange for reduced bandwidth usage.

To opt out of compression, use the DISABLED mode. Be aware that disabling compression might change CPU usage or throughput/latency, since the cluster must decompress each document before sending it to the connector.

Regardless of the compression mode you select, documents are published to the Kafka topic in uncompressed form.

That’s all for now. To download the Kafka connector and kick the tires, check out the Quickstart Guide. If you have any questions, come visit the Kafka connector category on the Couchbase Forums!


Posted by David Nault

David Nault writes code at Couchbase, where he works on the SDK & Connectors team.

Leave a reply