[MB-7199] Couchbase server can't handle hundreds of view queries with unlimited number of results at the same time

Wed, 27 Mar 2013 17:00:45 -0500

Cluster: 6 nodes
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43

Build # 2.0.0-1952 with 16 erlang schedulers
each nodes with 390GB SSD drive, 32GB RAM

2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.

I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
    capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
    url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
                                                design_doc_name, type_,
                                                view_name)
    headers = {'Content-Type': 'application/json',
               'Authorization': 'Basic %s' % authorization,
               'Accept': '*/*'}
     req = urllib2.Request(url, headers = headers)

Then the UI becomes unresponsive.
Pay attention to the following stats:

1st is the erlang scheduler on one of the nodes during query happens:

(ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
#Fun<erl_eval.6.80247286>
1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}

2nd is the top stats about beam.smp:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp

24G memory usage. And the CPU% is always above 350%