Couchbase .NET client library

Hangs after threads are aborted

Details

  • Type: Bug Bug
  • Status: In Progress In Progress
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 1.1.6
  • Fix Version/s: 1.2.7
  • Component/s: library
  • Labels:
    None

Description

In our production system thread occasionally get aborted to prevent long-running tasks from using resources when the user is no longer interested in the results. We are seeing these kinds of errors in the log:

ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Could not init pool.
and
ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Failed to reset an acquired socket.

Also, we are seeing hangs in MessageStreamListener.cs, line 400:
while ((line = reader.ReadLine()) != null)

This will occasionally hang indefinitely waiting for data to come in to the stream when there is no more data coming in.
I have created a reproduction application that creates a bunch of threads in an "abortable" thread pool (which just allows you to abort the threads, instead of being a black box). Starting the application will print a "." for every thread started, a "!" for every thread stopped and an "x" for every thread aborted. The thread itself just does a simple set/get. It typically hangs after about 15-20 seconds, on this ReadLine statement.

You can find the repro attached to this ticket.

This happens with Couchbase server 1.8.1 and the latest client code from Github.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Ingenthron added a comment -
Can you see if this is reproducible with current server/client?
Show
Matt Ingenthron added a comment - Can you see if this is reproducible with current server/client?
Hide
Permalink
Saakshi Manocha added a comment -
Cluster version - 2.0.1-170
Client - latest from Github

Please find the attached program HangRepro-NCBC-111.zip that I used to reproduce the scenario.
The output is "output-with-using-statement.txt" (attached herewith)
The program hangs after 15-20 seconds.

Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this:

while (true)
                {
                    Console.Write(".");
                    var client = new CouchbaseClient(section);
                    // Start a thread that just does a simple add/get and then stops
                    var wi = AbortableThreadPool.QueueUserWorkItem(_ =>
                                                   {
                                                       client.Store(StoreMode.Add, "somekey", "somevalue");
                                                       var someValue = client.Get<string>("somekey");
                                                       if (someValue != "somevalue") throw new InvalidOperationException();
                                                       Console.Write("!");
                                                   });

                    // Maybe kill the thread
                    if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);

                    // Wait a bit
                    Thread.Sleep((int)Math.Floor(rnd.NextDouble() * 10.0));

                    // Maybe kill the thread
                    if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);
                }

With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine.
The output is "output-without-using-statement.txt" (attached herewith)
Show
Saakshi Manocha added a comment - Cluster version - 2.0.1-170 Client - latest from Github Please find the attached program HangRepro- NCBC-111 .zip that I used to reproduce the scenario. The output is "output-with-using-statement.txt" (attached herewith) The program hangs after 15-20 seconds. Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this: while (true)                 {                     Console.Write(".");                     var client = new CouchbaseClient(section);                     // Start a thread that just does a simple add/get and then stops                     var wi = AbortableThreadPool.QueueUserWorkItem(_ =>                                                    {                                                        client.Store(StoreMode.Add, "somekey", "somevalue");                                                        var someValue = client.Get<string>("somekey");                                                        if (someValue != "somevalue") throw new InvalidOperationException();                                                        Console.Write("!");                                                    });                     // Maybe kill the thread                     if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);                     // Wait a bit                     Thread.Sleep((int)Math.Floor(rnd.NextDouble() * 10.0));                     // Maybe kill the thread                     if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);                 } With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine. The output is "output-without-using-statement.txt" (attached herewith)
Hide
Permalink
Saakshi Manocha added a comment -
@Roy: Could you please try again without the "using" statement, and let me know if it helps!
Show
Saakshi Manocha added a comment - @Roy: Could you please try again without the "using" statement, and let me know if it helps!

People

  • Assignee:
    Saakshi Manocha
    Reporter:
    roy.jacobs
Vote (0)
Watch (4)

Dates

  • Created:
    Updated: