System.ObjectDisposedException thrown by ThresholdLoggingTracer

.NET SDK
1

I am new to this forum so my apologies if the post does not look right…

Using CouchbaseNetClient 2.7.13
.NET Core 2.1
Couchbase Enterprise 6.0.1 /2037

During load testing/stress testing of our microservices, intermittently we see System.ObjectDisposedException thrown by the SDK.
These are thrown from various API calls (no preference) such as InsertAsync, GetDocumentAsync, ExistsAsync
And the stack looks like this:

System.ObjectDisposedException: The collection has been disposed. Object name: 'BlockingCollection'. 
at System.Collections.Concurrent.BlockingCollection`1.CheckDisposed() 
at System.Collections.Concurrent.BlockingCollection`1.TryAddWithNoTimeValidation(T item, Int32 millisecondsTimeout, CancellationToken cancellationToken) 
at Couchbase.Tracing.ThresholdLoggingTracer.ReportSpan(Span span) 
at OpenTracing.Util.AsyncLocalScope.Dispose() 
at Couchbase.Core.Buckets.CouchbaseRequestExecuter.SendWithRetryAsync[T](IOperation`1 operation, TaskCompletionSource`1 tcs, CancellationTokenSource cts) 
at Couchbase.CouchbaseBucket.InsertAsync[T](IDocument`1 document, TimeSpan timeout) 
at Couchbase.CouchbaseBucket.InsertAsync[T](IDocument`1 document, TimeSpan timeout)

The BlockingCollection in question is in Couchbase/Tracing/ThresholdLoggingTracer

private readonly BlockingCollection<SpanSummary> _queue = new BlockingCollection<SpanSummary>(1000);

and the method that triggers the exception is this:

internal void ReportSpan(Span span)
{
    if (span.IsRootSpan && !span.ContainsIgnore)
    {
        var summary = new SpanSummary(span);
        if (IsOverThreshold(summary))
        {
            _queue.Add(summary);
        }
    }
}

The place where the member _queue is disposed in this class is here:

public void Dispose()
{
    _source?.Cancel();

    if (_queue != null)
    {
        _queue.CompleteAdding();
        while (_queue.Any())
        {
            _queue.TryTake(out _);
        }

        _queue.Dispose();
    }
}

So it looks like more defensive programming should be applied? such as setting _queue to null after Dispose() and checking against null value or perhaps implementing the full Dispose pattern as recommended here: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/implementing-dispose

2

Hey @gabrielm, thanks for posting!

Yes, it looks like an issue in our threshold logging tracer. It’s there just to get telemetry about what is not behaving within normal operating parameters in the log.

As a workaround for now, you may just want to disable that. It should be covered in the documentation.

I’ve filed NCBC-2194 to track this. You should be able to watch it there too.

3

@ingenthr - Thank you so much; such a quick response :slight_smile: Looking forward to the fix, and I think I know what you mean – you must mean these properties in ClientConfiguration: OperationTracingEnabled, OperationTracingServerDurationEnabled;

4

I’ve made a fix to check the blocking collection has not been marked as complete before trying to add a new span for reporting: http://review.couchbase.org/118496

1 Like