.NET SDK - Infinite query execution

[Felix_123]

I have a problem using the Couchbase .NET SDK when executing a read only query.
The query timeout in the ClusterSettings is set to 15 seconds.

Couchbase Version: 7.2.2
Couchbase .NET SDK Version: 3.4.13
.NET Version: 6.0

Code:

_logger.LogInformation("Fetch data ...");

var statement = "SELECT ...";
var bucket = await _bucketProvider.GetBucketAsync().ConfigureAwait(false);
var result = await bucket.Cluster.QueryAsync<MyObject>(statement,
    options => options
                      .ScanConsistency(QueryScanConsistency.RequestPlus)
                      .AdHoc(false)
                      .Metrics(true)
                      .Readonly(true)).ConfigureAwait(false);

// DOES NOT GET PRINTED
_logger.LogInformation("Unreachable");

When the code executes the first log line is printed and after that nothing happens. It seems like the query timeout does not work properly.
The problem occurs after around 2 hours when running in a Docker Container with a memory limit of 500Mb.

Can someone please help me to identify the Problem.

[mreiche]

Can you please show all your code include the code that creates the cluster connection?

[Felix_123]

I am using Dependency Injection to create the Cluster connection.

It is running in a BackgroundService in a 30 seconds interval.

Program.cs

var cbSettings = builder.Configuration
    .GetSection(CouchbaseSettings.AbsoluteConfigKey)
    .Get<CouchbaseSettings>()
    ?? throw new NullReferenceException(nameof(CouchbaseSettings));

builder.Services.AddCouchbase(co =>
{
    co.ConnectionString = cbSettings.Hostname;
    co.UserName = cbSettings.UserName;
    co.Password = cbSettings.Password;
    co.Serializer = new CbSerializer();
    co.Logging = new LoggerFactory().AddSerilog(serilogLogger);
    co.RetryStrategy = new FailFastRetryStrategy();

    co.EnableConfigPolling = true;
    co.ConfigPollInterval = TimeSpan.FromSeconds(cbSettings.ConfigPollIntervalInSeconds);
    co.BootstrapPollInterval = TimeSpan.FromSeconds(cbSettings.BootstrapPollIntervalInSeconds);
    co.TracingOptions = new TracingOptions()
    {
        Enabled = cbSettings.EnableTracing
    };

    co.EnableTls = cbSettings.EnableTls;
    co.MaxHttpConnections = cbSettings.MaxHttpConnections;
    co.HttpConnectionLifetime = TimeSpan.FromMinutes(cbSettings.HttpConnectionLifetimeInMinutes);
    co.IdleHttpConnectionTimeout = TimeSpan.FromMinutes(cbSettings.IdleHttpConnectionTimeoutInMinutes);
    co.HttpIgnoreRemoteCertificateMismatch = cbSettings.HttpIgnoreRemoteCertificateMismatch;

    co.EnableTcpKeepAlives = cbSettings.EnableTcpKeepAlives;
    co.TcpKeepAliveInterval = TimeSpan.FromMinutes(cbSettings.TcpKeepAliveIntervalInMinutes);

    co.KvTimeout = TimeSpan.FromSeconds(cbSettings.KvTimeoutInSeconds);
    co.KvConnectTimeout = TimeSpan.FromSeconds(cbSettings.KvConnectTimeoutInSeconds);
    co.IdleKvConnectionTimeout = TimeSpan.FromSeconds(cbSettings.IdleKvConnectionTimeoutInSeconds);
    co.KvIgnoreRemoteCertificateNameMismatch = cbSettings.KvIgnoreRemoteCertificateNameMismatch;

    co.QueryTimeout = TimeSpan.FromSeconds(15);
});
builder.Services.AddCouchbaseBucket<IMyBucketProvider>(cbSettings.Bucket);

MyRepository.cs

private readonly ILogger<MyRepository> _logger;
private readonly IMyBucketProvider _bucketProvider;

public MyRepository(
    ILogger<MyRepository> logger,
    IMyBucketProvider bucketProvider)
{
    _logger = logger;
    _bucketProvider = bucketProvider;
}

public async Task<IReadOnlyList<MyObject>> GetAllData()
{
    _logger.LogInformation("Fetch data ...");

    var statement = "SELECT ...";
    var bucket = await _bucketProvider.GetBucketAsync().ConfigureAwait(false);
    var result = await bucket.Cluster.QueryAsync<MyObject>(statement,
        options => options
                          .ScanConsistency(QueryScanConsistency.RequestPlus)
                          .AdHoc(false)
                          .Metrics(true)
                          .Readonly(true)).ConfigureAwait(false);
    
    // DOES NOT GET PRINTED
    _logger.LogInformation("Unreachable");

    var data = new List<MyObject>();
    await foreach (var row in result.ConfigureAwait(false))
    {
        data.Add(row);
    }

    _logger.LogInformation("Fetched:  {Count} data", data.Count);

    return data.AsReadOnly();
}

[mreiche]

The default query timeout is 75 seconds, so even if your builder query timeout of 15 seconds wasn’t working, the query would timeout in 75 seconds. You could try executing the query “SELECT 1” just to test the flow of your app.

[btburnett3]

I’ve reviewed the code and I don’t see any problem that would cause your symptoms at a glance. It does appear that it could take a bit longer than the timeout if there are retries, but no more than 2x the configured timeout if a retry were to start immediately before the 15 second timeout was reached.

That said, I could easily be missing some subtle corner case. The SDK includes SourceLink, so if you disable Just My Code in Visual Studio you should be able to step through the SDK source and perhaps find where it’s hanging?