Hi Syed,
Thanks for using our Kafka connector!
the black hole topic contains sample messages where the key is “ignored-208,” and the value is empty.
Yes, this is the expected behavior. One design goal for the black hole topic was to minimize the amount of data stored in the topic. Using a fixed pool of record keys means more disk space can be reclaimed when the topic is compacted.
It sounds like you want to know which messages were sent to the black hole. The connector only sends a document to the black hole topic if:
- It came from a Couchbase system metadata collection. These documents are irrelevant in most cases, so the connector drops them by default. If you don’t want to drop them, see the documentation for the
couchbase.event.filterconnector config option.
OR
- A user’s custom
FilterorSourceHandlerimplementation decided the message should be dropped (filtered out).
In other words, the document almost always ended up in the black hole because the user wrote custom code to send it there. One way to know which documents got sent there would be to modify that custom user code to do additional logging.
If you’re trying to diagnose a problem in a dev environment, another option is to enable document lifecycle logging and look for SOURCE_OFFSET_UPDATE_COMMITTED_TO_BLACK_HOLE_TOPIC milestones in the connector log. These log messages contain the document ID and other metadata.
If neither of these suggestions work for you, please let us know more about the problem you’re trying to solve, and maybe we can offer advice tailored to your use case.
Thanks,
David