Couchbase Server

rebalance performance regression in 2.0.1 vs 2.0.0 (apparently only under load)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.0.1
  • Fix Version/s: None
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    linux

Description

Recent benchmarks: http://dashboard.hq.couchbase.com/litmus/dashboard/

Summary:

Rebalance-in ( 2-4 nodes), 7M items, mixed workload:

ec2: 2.0.0-1976 (RTM) took 1687 sec while 2.0.1-116 took 3581 sec
thor (data center physical machines): 2.0.0-1976 (RTM) took 901 sec while 2.0.1-123 took 1553 sec

Rebalance-out (4-2 nodes), 7M items, mixed workload:

ec2: 2.0.0-1976 (RTM) took 2854 sec while 2.0.1-116 took 5075 sec
thor (data center physical machines): 2.0.0-1976 (RTM) took 1142 sec while 2.0.1-123 took 2773 sec

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Ronnie Sun added a comment -
diags:

thor (data center machines), build 2.0.1-123-rel:

reb-1: http://172.23.96.10:8080/job/thor-parent/112/

reb-1-out: http://172.23.96.10:8080/job/thor-parent/110/
Show
Ronnie Sun added a comment - diags: thor (data center machines), build 2.0.1-123-rel: reb-1: http://172.23.96.10:8080/job/thor-parent/112/ reb-1-out: http://172.23.96.10:8080/job/thor-parent/110/
Steve Yen made changes -
Field Original Value New Value
Summary rebalance regression rebalance performance regression in 2.0.1 vs 2.0.0
Hide
Permalink
Pavel Paulau added a comment -
+ summary of test view views:

https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0
Show
Pavel Paulau added a comment - + summary of test view views: https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0
Hide
Permalink
Aleksey Kondratenko added a comment -
Folks, I still see no results from 2.0.0 with +A. May I insist on having some ?
Show
Aleksey Kondratenko added a comment - Folks, I still see no results from 2.0.0 with +A. May I insist on having some ?
Hide
Permalink
Aleksey Kondratenko added a comment -
Alternatively we can try 2.0.1 _without_ +A
Show
Aleksey Kondratenko added a comment - Alternatively we can try 2.0.1 _without_ +A
Hide
Permalink
Aleksey Kondratenko added a comment -
Did some runs comparing 2.0.0 and latest branch-2.0.1 and I've found 2.0.1 to be _faster_.

All my data fits in page cache plus I didn't send any mutations during rebalance. Plus I've allowed all nodes to use just one (same) CPU core. I'll rerun with binding different node to different cores.
Show
Aleksey Kondratenko added a comment - Did some runs comparing 2.0.0 and latest branch-2.0.1 and I've found 2.0.1 to be _faster_. All my data fits in page cache plus I didn't send any mutations during rebalance. Plus I've allowed all nodes to use just one (same) CPU core. I'll rerun with binding different node to different cores.
Hide
Permalink
Farshid Ghods added a comment -
assigning this back to Ronnie as Alk is expecting results for 2.0.0 with +A
Show
Farshid Ghods added a comment - assigning this back to Ronnie as Alk is expecting results for 2.0.0 with +A
Farshid Ghods made changes -
Assignee Aleksey Kondratenko [ alkondratenko ] Ronnie Sun [ ronnie ]
Hide
Permalink
Pavel Paulau added a comment -
Hi Aleksey,

I added first results for reb-out with +A, it shows >1.5x regression (see link above). I'm gathering more results but apparently it takes time.

Anyway, I believe KV results with +A will be even more helpful.
Show
Pavel Paulau added a comment - Hi Aleksey, I added first results for reb-out with +A, it shows >1.5x regression (see link above). I'm gathering more results but apparently it takes time. Anyway, I believe KV results with +A will be even more helpful.
Hide
Permalink
Aleksey Kondratenko added a comment -
No. Let's have _separate_ bug for rebalance with views.
Show
Aleksey Kondratenko added a comment - No. Let's have _separate_ bug for rebalance with views.
Aleksey Kondratenko made changes -
Assignee Ronnie Sun [ ronnie ] Aleksey Kondratenko [ alkondratenko ]
Hide
Permalink
Aleksey Kondratenko added a comment -
Given that I've tried rebalance myself and saw no regression (in fact speedup) in 2.0.1 I think we can start assuming that problem only occurs if rebalance is performed under load.

BTW, I've also confirmed personally that erlang core performance was not regressed in 2.0.1 (i.e. we have added -fno-strict-aliasing to CFLAGS).
Show
Aleksey Kondratenko added a comment - Given that I've tried rebalance myself and saw no regression (in fact speedup) in 2.0.1 I think we can start assuming that problem only occurs if rebalance is performed under load. BTW, I've also confirmed personally that erlang core performance was not regressed in 2.0.1 (i.e. we have added -fno-strict-aliasing to CFLAGS).
Aleksey Kondratenko made changes -
Summary rebalance performance regression in 2.0.1 vs 2.0.0 rebalance performance regression in 2.0.1 vs 2.0.0 (apparently only under load)
Hide
Permalink
Aleksey Kondratenko added a comment -
Fix is merged as part of chain ending at: http://review.couchbase.org/24067

I've found that my original plan to try to end serial phase of vbucket move at end of backfill helps a lot.
Show
Aleksey Kondratenko added a comment - Fix is merged as part of chain ending at: http://review.couchbase.org/24067 I've found that my original plan to try to end serial phase of vbucket move at end of backfill helps a lot.
Aleksey Kondratenko made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Ronnie Sun
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: