We are using the Couchbase Kafka Connector version 4.2.6. I have configured the “couchbase.black.hole.topic” for my connector. After configuring it, I noticed that the black hole topic contains sample messages where the key is “ignored-208,” and the value is empty.
I understand that this topic is used to track offset processing. Is this the expected behavior, or is there a way to retrieve completely ignored documents or at least include an ID in the value column to identify which documents are being ignored?
Currently, the “ignored-Vbucket” key does not provide meaningful information to help identify the missing documents. If this is not the expected behavior, could you please advise on how this can be achieved?
the black hole topic contains sample messages where the key is “ignored-208,” and the value is empty.
Yes, this is the expected behavior. One design goal for the black hole topic was to minimize the amount of data stored in the topic. Using a fixed pool of record keys means more disk space can be reclaimed when the topic is compacted.
It sounds like you want to know which messages were sent to the black hole. The connector only sends a document to the black hole topic if:
A user’s custom Filter or SourceHandler implementation decided the message should be dropped (filtered out).
In other words, the document almost always ended up in the black hole because the user wrote custom code to send it there. One way to know which documents got sent there would be to modify that custom user code to do additional logging.
If you’re trying to diagnose a problem in a dev environment, another option is to enable document lifecycle logging and look for SOURCE_OFFSET_UPDATE_COMMITTED_TO_BLACK_HOLE_TOPIC milestones in the connector log. These log messages contain the document ID and other metadata.
If neither of these suggestions work for you, please let us know more about the problem you’re trying to solve, and maybe we can offer advice tailored to your use case.