Couchbase Server

memcached should dynamically adjust the number of worker threads

Details

  • Type: Improvement Improvement
  • Status: In Progress In Progress
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: bucket-engine
  • Security Level: Public

Description

4 threads is probably not ideal for a 24 core system ;)

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Trond Norbye added a comment -
http://review.couchbase.org/#/c/24294/
Show
Trond Norbye added a comment - http://review.couchbase.org/#/c/24294/
Hide
Permalink
Dipti Borkar added a comment -
what's the impact on the rest of the system?

for indexing and XDCR, we need a lot of cpu now.

Show
Dipti Borkar added a comment - what's the impact on the rest of the system? for indexing and XDCR, we need a lot of cpu now.
Hide
Permalink
Trond Norbye added a comment -
How many cores do you need for indexing and xdcr?
Show
Trond Norbye added a comment - How many cores do you need for indexing and xdcr?
Hide
Permalink
Dipti Borkar added a comment -
1 core per design doc. XDCR is also CPU intensive.
both are very CPU intensive operations.
Show
Dipti Borkar added a comment - 1 core per design doc. XDCR is also CPU intensive. both are very CPU intensive operations.
Hide
Permalink
Trond Norbye added a comment -
Do you have a formula on how I can calculate this at runtime??? the current model isn't very good since in order to change it I have to move the memcached binary away and replace it with a wrapper script that adds the -t parameter to the command arguments before starting it.

I guess we could add an "advanced settings tab" somewhere in our web UI where we could let the user override the number of threads?
Show
Trond Norbye added a comment - Do you have a formula on how I can calculate this at runtime??? the current model isn't very good since in order to change it I have to move the memcached binary away and replace it with a wrapper script that adds the -t parameter to the command arguments before starting it. I guess we could add an "advanced settings tab" somewhere in our web UI where we could let the user override the number of threads?
Hide
Permalink
Dipti Borkar added a comment -
"I guess we could add an "advanced settings tab" somewhere in our web UI where we could let the user override the number of threads?"

Yes, making it parameterized is probably the way to go for now. Typically we have been doing this via the rest API and adding it to /internalSettings.
Does this need server restart?

Also, do we understand benefits from a performance perspective when # of threads are increased?
Show
Dipti Borkar added a comment - "I guess we could add an "advanced settings tab" somewhere in our web UI where we could let the user override the number of threads?" Yes, making it parameterized is probably the way to go for now. Typically we have been doing this via the rest API and adding it to /internalSettings. Does this need server restart? Also, do we understand benefits from a performance perspective when # of threads are increased?
Hide
Permalink
Trond Norbye added a comment -
The node (memcached process) needs to be restarted in order for the change to take effect.

The "easy" way to see if you should increase the number is to check if all of your threads is busy. using top on linux you can toggle the "threads" mode and if the 4 worker threads in memcached is ~80% busy you might want to increase the number of threads. (saying 80% since I don't know how top reports blocking for resources etc). If you're already saturating the network (or the cpu core that is handling network interrupts) there isn't a point of increasing the number of threads (unless the os supports fan'ing out the network interrupts to other cores).

I guess it boils down to: Do we want to set a cap on the maximum performance the user may use for KV access and leave resources unused (if people aren't using views and xdcr at all). And the next question is: should we rather allow memcached to use more worker threads and then boost the priority in the OS for XDCR and index generation (I would still expect that to be somewhat IO intensive and not CPU intensive given that it has to read the stuff off disk to process it?) to give those processes bigger shares at the OS scheduling of the threads to the core..
Show
Trond Norbye added a comment - The node (memcached process) needs to be restarted in order for the change to take effect. The "easy" way to see if you should increase the number is to check if all of your threads is busy. using top on linux you can toggle the "threads" mode and if the 4 worker threads in memcached is ~80% busy you might want to increase the number of threads. (saying 80% since I don't know how top reports blocking for resources etc). If you're already saturating the network (or the cpu core that is handling network interrupts) there isn't a point of increasing the number of threads (unless the os supports fan'ing out the network interrupts to other cores). I guess it boils down to: Do we want to set a cap on the maximum performance the user may use for KV access and leave resources unused (if people aren't using views and xdcr at all). And the next question is: should we rather allow memcached to use more worker threads and then boost the priority in the OS for XDCR and index generation (I would still expect that to be somewhat IO intensive and not CPU intensive given that it has to read the stuff off disk to process it?) to give those processes bigger shares at the OS scheduling of the threads to the core..
Hide
Permalink
Dipti Borkar added a comment -
For the ns_server rest API, please work with Anil who can prioritize it with the ns_server folks
Show
Dipti Borkar added a comment - For the ns_server rest API, please work with Anil who can prioritize it with the ns_server folks

People

  • Assignee:
    Trond Norbye
    Reporter:
    Trond Norbye
Vote (0)
Watch (4)

Dates

  • Created:
    Updated: