Couchbase java sdk - Do not log query

[himanshu.mps]

Hi,

We are using couchbase 3 java SDK and we do not wish to log the query in case of any exception or threshold logging. Currently the logs show this:

com.couchbase.client.core.error.AmbiguousTimeoutException: QueryRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xd14d3b1700000001","idempotent":false,"lastDispatchedTo":"test50505.phx.com","reason":"TIMEOUT","requestId":23,"requestType":"QueryRequest","retried":0,"service":{"operationId":"abade0ed-fa7e-48ee-945a-c174dd3d61b7","statement":"select monocle.* from monocle use keys 'BOOKMARK::PREF::test8' WHERE type='fax';","type":"query"},"timeoutMs":5000,"timings":{"totalMicros":5009613}}

Can you please let us know what can be done to disable query logging?

Thanks,
Himanshu

[david.nault]

Hi Himanshu,

The SDK has a log redaction feature that tags potentially sensitive information like query statements, so they can be obfuscated before sharing or archiving logs. Would enabling log redaction resolve the issue you’re facing?

Thanks,
David

[Utkal]

Hi David,
Using that feature basically adds tag to the query statement. But it doesn’t redact it.

Is there any way we could directly redact those part on the fly when it gets printed on our application logs?

[mreiche]

You’ll need to supply your own logger that removes the redacted portions:

		LogRedaction.setRedactionLevel(RedactionLevel.PARTIAL);
		LoggingEventConsumer.Logger logger = new MySlf4JLogger("com.couchbase");
		ClusterEnvironment env = ClusterEnvironment.builder()
			.loggerConfig(l -> l.customLogger(logger))

import com.couchbase.client.core.cnc.LoggingEventConsumer;
import com.couchbase.client.core.logging.RedactableArgument;
import com.couchbase.client.core.util.CbCollections;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

import java.util.Map;
import java.util.logging.Level;
import java.util.stream.Collectors;

public class MySlf4JLogger implements  LoggingEventConsumer.Logger {
  private final org.slf4j.Logger logger;

  MySlf4JLogger(String name) {
    this.logger = LoggerFactory.getLogger(name);
  }

  public String getName() {
    return this.logger.getName();
  }

  public boolean isTraceEnabled() {
    return this.logger.isTraceEnabled();
  }

  public void trace(String msg) {
    this.logger.trace(redact(msg));
  }

  public void trace(String msg, Throwable t) {
    this.logger.trace(redact(msg), t);
  }

  public boolean isDebugEnabled() {
    return this.logger.isDebugEnabled();
  }

  public void debug(String msg) {
    this.logger.debug(redact(msg));
  }

  public void debug(String msg, Throwable t) {
    this.logger.debug(redact(msg), t);
  }

  public boolean isInfoEnabled() {
    return this.logger.isInfoEnabled();
  }

  public void info(String msg) {
    this.logger.info(redact(msg));
  }

  public void info(String msg, Throwable t) {
    this.logger.info(redact(msg), t);
  }

  public boolean isWarnEnabled() {
    return this.logger.isWarnEnabled();
  }

  public void warn(String msg) {
    this.logger.warn(redact(msg));
  }

  public void warn(String msg, Throwable t) {
    this.logger.warn(redact(msg), t);
  }

  public boolean isErrorEnabled() {
    return this.logger.isErrorEnabled();
  }

  public void error(String msg) {
    this.logger.error(redact(msg));
  }

  public void error(String msg, Throwable t) {
    this.logger.error(redact(msg), t);
  }

  private String redact(String msg){
    return msg.replaceAll("<ud>.*</ud>", "<redacted/>");
  }

  @Override
  public void attachContext(Map<String, Object> context) {
    if (!CbCollections.isNullOrEmpty(context)) {
      MDC.setContextMap((Map)context.entrySet().stream().collect(Collectors.toMap((e) -> {
        return RedactableArgument.redactUser(e.getKey()).toString();
      }, (e) -> {
        Object v = e.getValue();
        return v == null ? "" : RedactableArgument.redactUser(v.toString()).toString();
      })));
    }
  }

  public void clearContext() {
    MDC.clear();
  }

  public org.slf4j.Logger getImplementation() {
    return this.logger;
  }

}

If you want to customize your Slf4J logger…

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d %5p %40.40c:%4L - %m%n</pattern>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="console"/>
    </root>

    <logger name="com.couchbase" level="debug"/>"

</configuration>```

2023-04-10 11:41:10,436 DEBUG com.couchbase: 41 - [com.couchbase.request][RequestRetryScheduledEvent] Request QueryRequest retry scheduled per RetryStrategy (Reason: BUCKET_OPEN_IN_PROGRESS) {“cancelled”:true,“completed”:true,“coreId”:“0xf7ae716e00000001”,“idempotent”:false,“reason”:“TIMEOUT”,“requestId”:4,“requestType”:“QueryRequest”,“retried”:1,“retryReasons”:[“BUCKET_OPEN_IN_PROGRESS”],“service”:{“bucket”:“daniel-wucmmh”,“operationId”:“null”,“scope”:“samples”,“statement”:“<redacted/>”,“type”:“query”},“timeoutMs”:1,“timings”:{“totalMicros”:7366}}

[david.nault]

You’ll need to supply your own logger that removes the redacted portions:

Note that the customLogger config option is not part of the public API, and might change without notice. An alternative using only public APIs would be to configure your logging framework to redact the messages. With log4j2, a custom LogEventPatternConverter might be the way to go.

We can also look into addressing the query statement redaction from the SDK side. We’ll need to run it by PM and think about the best way to implement it.

Thanks,
David

[david.nault]

Another thought: If you’re concerned about redacting the statement because it might have sensitive user data, you might be able to prevent the sensitive bits from being logged by treating them as query parameters:

QueryResult result = clusterOrScope.query(
  "select monocle.* from monocle use keys ? WHERE type=?",
  QueryOptions.queryOptions()
    .parameters(JsonArray.from("BOOKMARK::PREF::test8", "fax"))
);

This way, the exception looks like:

Exception in thread "main" com.couchbase.client.core.error.AmbiguousTimeoutException: QueryRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x963912ee00000001","idempotent":false,"reason":"TIMEOUT","requestId":2,"requestType":"QueryRequest","retried":1,"retryReasons":["BUCKET_OPEN_IN_PROGRESS"],"service":{"operationId":"null","statement":"select monocle.* from monocle use keys ? WHERE type=?","type":"query"},"timeoutMs":1,"timings":{"totalMicros":10727}}

As a bonus, parameters have the benefit of being resistant to SQL injection.

Thanks,
David