Good day Everybody. I come with this issue. I would like to know more about how max_parellelism property works over query execution in Couchbase. Based on previous information I’ve read, this property has a limit on Community Edition (4 as max). When I query the current value in that edition, it is setted to 1 by default
{“completed-limit”:4000,“completed-threshold”:1000,“cpuprofile”:"",“debug”:false,“keep-alive-length”:16384,“loglevel”:“INFO”,“max-parallelism”:1,“memprofile”:"",“pipeline-batch”:16,“pipeline-cap”:512,“pretty”:true,“request-size-cap”:67108864,“scan-cap”:0,“servicers”:24,“timeout”:0}
What does that mean?.. That max_parellelism = 1 runs one thread at a time and set it to 4 will run 4 threads instead??
When query service started you see the following message in query.log _time=2018-07-18T14:08:45.042-07:00 _level=INFO _msg=cbq-engine started max-concurrency=8 pipeline-cap=512 timeout=0s datastore=http://127.0.0.1:8091 scan-cap=512 pipeline-batch=16 request-cap=256 version=2.0.0-N1QL servicers=32 plus-servicers=128 max-index-api=3 n1ql_feat_cntrl=0 loglevel=INFO request-size-cap=67108864
max-concurrency tells how any logical CPUs that query engine going to use on the machine. For CE this value will be maximum 4, For EE this value is logical CPU’s on the machine.
servicers = 4*max-concurrency -- Number of concurrent unbounded queries processed
plus-servicers= 16*max-concurrency -- Number of concurrent request_plus queries are processed
max-parallelism is with in the query. Some operations of queries can be run in parallel (like grouping,aggregation, filter, In EXPLAIN this indicated by “#operator”: “Parallel”) In those situations it will be used that many threads. setting high value will consumes more cpu and memory
Thanks @vsr1 for you response. So as you explain it, this value is a factor that you multiply by the limit (in that case 4 for unbound queries and 16 for request_plus queries in CE) . That operation return the max value I can reach for parallelism into the platform. I suppose that in EE, that factor directly turns into the number of cores my server has and that I’ve setted explicitly, right??
Moreover, if I change the factor (in CE) with for example 4 (4*4=16). In that stage, Couchbase will set the max to the predefined limit (4 instead of 16). Doesn’t it?
Assume your machine has 8 CPUs.
CE uses only 4 CPUS, EE uses all 8 CPUS. This has been displayed by max-concurrency in query.log
Lets talk about CE
Now you have 50 simultaneous clients issuing unbounded requests
Only 4 times 4 = 16 (indicated by servicers=16) queries are executed concurrently all others are queued.
Now if max_parallelism=3 each query as indicated in EXPLAIN “#operator”: “Parallel” code is done by that many parallel threads.
Thanks @vsr1. With that explanation I have it clear about how Couchbase handle concurrent processes for a specific number of CPUs. One last question. If you say that “servicers” prop indicates the total threads that run over available CPUs (4 in CE), and considering that its property value defaults to 32 (based on couchbase doc). Does this value dictates a limit for both CE and EE??.