Couchbase events in apache Kafka without base64 encoding

How to send content of couchbase events in apache Kafka without base64 encoding?

@david.nault can you please assist ?

Hi Ankita,

For version 3.4 of the connector,

dcp.message.converter.class=com.couchbase.connect.kafka.handler.source.RawJsonSourceHandler
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

Readers from the future should know that in version 4.x the property name will change from dcp.message.converter.class to couchbase.source.handler.

For more details and options, see the Publishing Raw JSON Messages section of the quickstart guide.

Thanks,
David

Thanks @david.nault for the reply and the message I am getting in Kafka is schemaless, I am looking for schema and content for payload in string format instead of base64 encoded.

Please help to understand the configurations or changes needed to do so.

Ankita

Hi Ankita,

The schema used by DefaultSchemaSourceHandler defines the content of the document as a byte array. Byte array fields are always Base64-encoded.

If you want to use Schemas but don’t want the Base64 encoding, you’ll need to write a custom SourceHandler that defines a schema that matches your Couchbase documents. The source handler will also need to convert the JSON document to a Kafka Connect Struct (copying each field of the JSON into the struct) (EDIT: I might have been wrong about having to use a Struct; it might be sufficient to convert the JSON to a Map<String,Object>.) You’ll need to do this for each type of document you want to publish.

The connector can’t do this automatically, because there’s no schema information in Couchbase.

If you want to walk that path, you can use the custom components example project as a template.

Thanks,
David