How can I filter out (or only include) certain fields in the event produced to kafka

Kafka Connector
1

I’d like if it makes sense to do this in the custom source handler.

I’m already using the CSH to filter for docs of specific types and to route to different topics depending on that doc type.

I am looking to remove a set of fields from the doc (depending on the type) before producing to the kafka topic.

What’s the right way to do that? Should I be doing that on the consuming side of things?

2

Hi Naftali,

Since you already have a custom source handler, you might find it easiest to remove the fields there, since you can run whatever Java code you want.

An alternative would be to configure a bunch of Single Message Transforms (SMTs) to remove fields, but I expect that could get really verbose – or might be impossible, since you want to remove different fields from different types of records.

Thanks,
David

3

Since you already have a custom source handler, you might find it easiest to remove the fields there, since you can run whatever Java code you want.

thank you.

I’m supposed to override the handle method, right?

is there an example you could point to?

4

I’m supposed to override the handle method, right?

Yes, that would work. Or the buildValue method if you think that would be less work. Either way, you’ll probably have to copy some code from the base class, since there’s not a great extension point just for transforming the document content. (Maybe that will change in the future.)

All of the examples are here.

1 Like