Frequent Timeouts and "Requests over Threshold" for SubdocGetRequests via Reactive Java SDK

Java SDK
,
1

Hello,

I’m attempting to write high-performing test comparing all the data in my old DB2 database with all the data in my new Couchbase database to ensure our sync process between the two continues to be successful. The DBAs have set up a job to export all the relevant DB2 data into files, so streaming thru those is very fast. However I’m having trouble reading quickly from couchbase using the SubDoc API and I’m not sure what the problem is.

My comparator test application is written using the Spring Webflux Reactor reactive Java framework, so I’m attempting to use Couchbase’s Reactive cluster object to fetch the relevant sections of the document I happen to be comparing at any given time. However, very soon after the test starts I’m getting a warning in the logs before my application inevitably hangs:

[com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found … “operation_name”:“SubdocGetRequest”… “service”:“kv”,“count”:1014

We’re using Couchbase Enterprise Edition 6.0.2 build 2413 - is the Subdoc API not quite ready for use, have I misconfigured my Java application, or do I need to spin up a KV node or modify my Couchbase cluster’s configuration in some way perhaps?

Here’s the configuration and service method I’m using for my Java application:

@Bean
public ReactiveCluster cluster(MyConfig config) {
    ClusterEnvironment environment = ClusterEnvironment.builder()
            .timeoutConfig(TimeoutConfig.builder()
                    .connectTimeout(Duration.ofSeconds(60L))
                    .disconnectTimeout(Duration.ofSeconds(60L))
                    .queryTimeout(Duration.ofSeconds(30L))
                    .kvTimeout(Duration.ofSeconds(30L))
                    .viewTimeout(Duration.ofSeconds(30L))
            )
            .build();
    ClusterOptions options = ClusterOptions.clusterOptions(
            config.getUsername(),
            config.getPassword())
            .environment(environment);
    Set<SeedNode> seedNodes = new HashSet<>();
    config.getBootstrapHosts().forEach(host -> seedNodes.add(
            SeedNode.create(host,
            Optional.of(11210),
            Optional.of(18091))));
    Cluster cluster = Cluster.connect(seedNodes, options);
    return cluster.reactive();
}

@Bean
public ReactiveBucket bucket(MyConfig config, ReactiveCluster cluster) {
    ReactiveBucket bucket = cluster.bucket(config.getBucket().getName());
    bucket.waitUntilReady(Duration.ofSeconds(60L))
            .block();
    return bucket;
}

public Mono<JsonObject> fetchSectionOfDocument(String documentId, String section) {
    return bucket.defaultCollection().lookupIn(documentId, Collections.singletonList(LookupInSpec.get(section)))
            .retryWhen(Retry.backoff(20L, Duration.ofMillis(500L)))
            .map(result -> result.contentAsObject(0))
            .onErrorReturn(PathNotFoundException.class, JsonObject.create());
}
2 Likes