Details
-
Type:
Bug
-
Status:
Open
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 2.0, 2.0.1, 2.0.2
-
Fix Version/s: 2.1
-
Component/s: ns_server, view-engine
-
Security Level: Public
-
Labels:
-
Environment:centos5.6, build 1952 with 16 scheduler
Description
Cluster: 6 nodes
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43
Build # 2.0.0-1952 with 16 erlang schedulers
each nodes with 390GB SSD drive, 32GB RAM
2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.
I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
design_doc_name, type_,
view_name)
headers = {'Content-Type': 'application/json',
'Authorization': 'Basic %s' % authorization,
'Accept': '*/*'}
req = urllib2.Request(url, headers = headers)
Then the UI becomes unresponsive.
Pay attention to the following stats:
1st is the erlang scheduler on one of the nodes during query happens:
(ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
#Fun<erl_eval.6.80247286>
1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}
2nd is the top stats about beam.smp:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp
24G memory usage. And the CPU% is always above 350%
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.42
10.6.2.43
Build # 2.0.0-1952 with 16 erlang schedulers
each nodes with 390GB SSD drive, 32GB RAM
2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec.
I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results:
capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT)
url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket,
design_doc_name, type_,
view_name)
headers = {'Content-Type': 'application/json',
'Authorization': 'Basic %s' % authorization,
'Accept': '*/*'}
req = urllib2.Request(url, headers = headers)
Then the UI becomes unresponsive.
Pay attention to the following stats:
1st is the erlang scheduler on one of the nodes during query happens:
(ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end.
#Fun<erl_eval.6.80247286>
1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0}
1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0}
1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0}
1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0}
1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0}
1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0}
1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0}
1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0}
1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0}
1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0}
1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0}
1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0}
1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0}
1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0}
1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0}
1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0}
1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0}
1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0}
1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0}
1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0}
1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0}
1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0}
1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0}
1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0}
1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0}
1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0}
1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0}
1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0}
1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0}
1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0}
1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0}
1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0}
1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0}
1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0}
1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0}
1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0}
1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0}
1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0}
1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0}
1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0}
1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0}
1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0}
1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0}
1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0}
2nd is the top stats about beam.smp:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp
24G memory usage. And the CPU% is always above 350%
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Chisheng Hong
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Description |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% erl_crash.dmp location: root:couchbase@10.3.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651 |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651 |
Chisheng Hong
made changes -
| Description |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/bin/erl_crash.dump.11-15-2012-17\:14\:38.651 |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/var/lib/couchbase/erl_crash.dump.11-15-2012-17\:14\:38.651 |
Tommie McAfee
made changes -
| Attachment | queries2.png [ 15835 ] |
Aleksey Kondratenko
made changes -
| Assignee | Aleksey Kondratenko [ alkondratenko ] | Aliaksey Artamonau [ Aliaksey Artamonau ] |
Aliaksey Artamonau
made changes -
| Assignee | Aliaksey Artamonau [ Aliaksey Artamonau ] | Tommie McAfee [ tommie ] |
Chisheng Hong
made changes -
| Description |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% erl_crash.dmp location: root:couchbase@10.6.2.37:/opt/couchbase/var/lib/couchbase/erl_crash.dump.11-15-2012-17\:14\:38.651 |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% |
Farshid Ghods
made changes -
| Assignee | Tommie McAfee [ tommie ] | Chisheng Hong [ Chisheng ] |
Chisheng Hong
made changes -
| Summary | Couchbase server can't handle view query with S 16:16 with build 1952 | Couchbase server can't handle view query with unlimited number of results |
| Assignee | Chisheng Hong [ Chisheng ] | Aliaksey Artamonau [ Aliaksey Artamonau ] |
| Environment | centos5.6 | centos5.6, build 1952 with 16 scheduler |
| Description |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. I don't put any limit on query results: capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT) url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket, design_doc_name, type_, view_name) headers = {'Content-Type': 'application/json', 'Authorization': 'Basic %s' % authorization, 'Accept': '*/*'} req = urllib2.Request(url, headers = headers) Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% |
Chisheng Hong
made changes -
| Description |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. I don't put any limit on query results: capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT) url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket, design_doc_name, type_, view_name) headers = {'Content-Type': 'application/json', 'Authorization': 'Basic %s' % authorization, 'Accept': '*/*'} req = urllib2.Request(url, headers = headers) Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% |
Cluster: 6 nodes
10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.42 10.6.2.43 Build # 2.0.0-1952 with 16 erlang schedulers each nodes with 390GB SSD drive, 32GB RAM 2 buckets created sasl and default. Start loading items with 8K creates per sec to each bucket. Then insert a ddoc with 2 views to each bucket. Then have 4 clients do query for the view with 120 reads per sec. I don't put any limit on query results and those queries are generated without waiting for previous ones finish showing the results: capiUrl = "http://%s:%s/couchBase/" % (cfg.COUCHBASE_IP, cfg.COUCHBASE_PORT) url = capiUrl + '%s/_design/%s/_%s/%s' % (bucket, design_doc_name, type_, view_name) headers = {'Content-Type': 'application/json', 'Authorization': 'Basic %s' % authorization, 'Accept': '*/*'} req = urllib2.Request(url, headers = headers) Then the UI becomes unresponsive. Pay attention to the following stats: 1st is the erlang scheduler on one of the nodes during query happens: (ns_1@10.6.2.37)5> F = fun (R) -> io:format("~p ~p~n", [latency:ts(now()), erlang:statistics(run_queues)]), timer:sleep(100), R(R) end. #Fun<erl_eval.6.80247286> 1353032384137 {11,104,2,0,8,11,0,0,0,0,0,0,0,0,0,0} 1353032384293 {4,65,103,7,2,20,0,0,0,0,0,0,0,0,0,0} 1353032384425 {3,7,4,25,21,3,0,0,0,0,0,0,0,0,0,0} 1353032384553 {23,17,50,6,6,0,0,0,0,0,0,0,0,0,0,0} 1353032384672 {16,28,92,15,65,42,0,0,0,0,0,0,0,0,0,0} 1353032384795 {6,4,47,15,1,0,0,0,0,0,0,0,0,0,0,0} 1353032384919 {1,11,86,59,56,55,0,0,0,0,0,0,0,0,0,0} 1353032385081 {54,49,30,44,33,11,0,0,0,0,0,0,0,0,0,0} 1353032385221 {15,47,10,45,9,31,0,0,0,0,0,0,0,0,0,0} 1353032385355 {46,2,72,89,28,4,0,0,0,0,0,0,0,0,0,0} 1353032385468 {11,1,8,26,0,2,0,0,0,0,0,0,0,0,0,0} 1353032385610 {7,23,7,14,20,13,0,0,0,0,0,0,0,0,0,0} 1353032385765 {7,85,11,16,0,12,0,0,0,0,0,0,0,0,0,0} 1353032385905 {9,29,28,2,3,26,0,0,0,0,0,0,0,0,0,0} 1353032386068 {48,112,142,31,12,25,0,0,0,0,0,0,0,0,0,0} 1353032386222 {11,40,28,36,5,9,0,0,0,0,0,0,0,0,0,0} 1353032386356 {64,53,4,5,7,34,0,0,0,0,0,0,0,0,0,0} 1353032386560 {0,2,45,2,0,89,0,0,0,0,0,0,0,0,0,0} 1353032386700 {50,18,83,4,0,35,0,0,0,0,0,0,0,0,0,0} 1353032386837 {0,18,3,2,17,4,0,0,0,0,0,0,0,0,0,0} 1353032386984 {2,10,11,6,0,4,0,0,0,0,0,0,0,0,0,0} 1353032387105 {1,5,12,2,0,64,0,0,0,0,0,0,0,0,0,0} 1353032387231 {22,67,58,5,19,7,0,0,0,0,0,0,0,0,0,0} 1353032387337 {17,1,38,33,7,1,0,0,0,0,0,0,0,0,0,0} 1353032387469 {5,5,48,27,2,18,0,0,0,0,0,0,0,0,0,0} 1353032387598 {2,50,47,88,41,8,0,0,0,0,0,0,0,0,0,0} 1353032387746 {2,55,16,35,1,12,0,0,0,0,0,0,0,0,0,0} 1353032387897 {3,29,98,0,5,19,0,0,0,0,0,0,0,0,0,0} 1353032388021 {29,50,147,0,5,3,0,0,0,0,0,0,0,0,0,0} 1353032388146 {15,3,30,3,46,2,0,0,0,0,0,0,0,0,0,0} 1353032388277 {53,8,50,1,10,14,0,0,0,0,0,0,0,0,0,0} 1353032388402 {2,19,45,0,6,2,0,0,0,0,0,0,0,0,0,0} 1353032388594 {17,123,2,0,29,4,0,0,0,0,0,0,0,0,0,0} 1353032388734 {35,92,0,3,40,70,0,0,0,0,0,0,0,0,0,0} 1353032388873 {2,10,22,5,18,17,0,0,0,0,0,0,0,0,0,0} 1353032389008 {112,84,15,0,1,0,0,0,0,0,0,0,0,0,0,0} 1353032389133 {102,57,0,25,3,23,0,0,0,0,0,0,0,0,0,0} 1353032389257 {44,55,28,5,36,49,0,0,0,0,0,0,0,0,0,0} 1353032389379 {4,40,3,48,2,48,0,0,0,0,0,0,0,0,0,0} 1353032389549 {24,161,24,38,16,21,0,0,0,0,0,0,0,0,0,0} 1353032389686 {54,25,12,23,7,98,0,0,0,0,0,0,0,0,0,0} 1353032389804 {79,33,20,2,3,46,0,0,0,0,0,0,0,0,0,0} 1353032389950 {90,0,25,13,45,56,0,0,0,0,0,0,0,0,0,0} 1353032390101 {59,10,17,1,37,54,0,0,0,0,0,0,0,0,0,0} 2nd is the top stats about beam.smp: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 676 couchbas 20 0 26.0g 24g 5128 S 663.9 77.7 369:01.85 beam.smp 24G memory usage. And the CPU% is always above 350% |
Chisheng Hong
made changes -
| Summary | Couchbase server can't handle view query with unlimited number of results | Couchbase server can't handle hundreds of view queries with unlimited number of results at the same time |
Filipe Manana
made changes -
| Assignee | Aliaksey Artamonau [ Aliaksey Artamonau ] | Aleksey Kondratenko [ alkondratenko ] |
Filipe Manana
made changes -
| Assignee | Aleksey Kondratenko [ alkondratenko ] | Chisheng Hong [ Chisheng ] |
Steve Yen
made changes -
| Labels | system-test | 2.0-release-notes system-test |
| Fix Version/s | 2.0.1 [ 10399 ] | |
| Fix Version/s | 2.0 [ 10114 ] |
Dipti Borkar
made changes -
| Fix Version/s | 2.0.2 [ 10418 ] | |
| Fix Version/s | 2.0.1 [ 10399 ] |
Filipe Manana
made changes -
| Attachment | erl_crash.dump.tgz [ 15916 ] | |
| Attachment | logs.tgz [ 15917 ] |
Filipe Manana
made changes -
| Component/s | ns_server [ 10019 ] |
Filipe Manana
made changes -
| Assignee | Chisheng Hong [ Chisheng ] | Aliaksey Artamonau [ Aliaksey Artamonau ] |
Jin Lim
made changes -
| Fix Version/s | 2.1 [ 10414 ] |
Jin Lim
made changes -
| Planned Start | (set to new fixed version's start date) | |
| Planned End | (set to new fixed version's start date) |
Maria McDuff
made changes -
| Fix Version/s | 2.0.2 [ 10418 ] |
Maria McDuff
made changes -
| Affects Version/s | 2.0.1 [ 10399 ] | |
| Affects Version/s | 2.0.2 [ 10418 ] |