Attempting to batch fetch files from couchbase through the .NET client, but my batchsize seems limited to around 20 items, whenever I try to increase the size I get “Couchbase.Core.Exceptions.RequestCanceledException” without have requested any cancelation on my end. Here is how I have my batch fetch method made.
public async Task<IEnumerable<T>> Get<T>(IEnumerable<string> keys)
{
var bucket = await _cluster.Value.Result.BucketAsync(Bucket);
var collection = bucket.DefaultCollection();
var tasks = keys.Select(async x =>
{
try
{
var result = await collection.GetAsync(x);
return result;
}
catch (DocumentNotFoundException)
{
return null;
}
}).ToList();
await Task.WhenAll(tasks);
return tasks.Where(x => x?.Result != null).Select(x => x.Result.ContentAs<T>());
}
I tried adjusting query settings on the server increased the Pipeline batch to 100, the Pipeline cap to 1024 and set MaxParallelism to 0 so it would use all available. any ideas that will help me increase batch size?
RequestCanceledException indicates that the request itself never completed; the socket connection itself was probably closed for some reason. I see your opening the bucket in the scope of a method, that is somewhat of an anti pattern, as we suggest opening the Cluster and Bucket at application startup and disposing them when the app shuts down. Enabling logging will likely make it easier to determine what exactly is going on, but I suspect something is simply closing the connections while in mid-flight.
serviceCollection.AddLogging(builder => builder
.AddFile("Logs/myapp-{Date}.txt", LogLevel.Debug)
);
var loggerFactory = serviceCollection.BuildServiceProvider().GetService<ILoggerFactory>();
var clusterOptions = new ClusterOptions().WithCredentials("Administrator", "password").WithLogging(loggerFactory);
var cluster = Cluster.ConnectAsync("couchbase://localhost", clusterOptions)
Also, you are awaiting before even calling await Task.WhenAll(tasks); so the tasks are actually completed before this time. You should remove the async and the await when you build your list of tasks. I see why your doing this, just pointing out that the behavior isn’t quite right.
Hey @CPike,
I have similar logic which you have here, mainly to ignore notfound error. How did you end up improving this block, so you don’t need individual async/away for each GetAsync?