Couchbase Server

XDCR: High CPU utilization on destination nodes

Details

  • Type: Bug Bug
  • Status: Reopened Reopened
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 2.0-beta-2
  • Fix Version/s: .next
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    CentOS 6.2, virtual clusters (1-to-1, 2-to-2, 4-to-4 nodes, 24GB RAM, 4 cores)

Description

Regardless cluster configuration CPU utilization on destination nodes is too high during initial replication.
Source cluster looks pretty idle at the same time.

See for details:
https://www.yammer.com/couchbase.com/#/Threads/show?threadId=200784413
https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdElKVEhla2NqTFJWdkpkcHFybGgtR3c#gid=4

Build 1550.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Junyi Xie added a comment -
With latest change, Ketaki saw in her test ( 3 source nodes -> 5 dest nodes) 50-60% CPU on destination clusters, without views/indexes running on this cluster, no additional load on this system.

Show
Junyi Xie added a comment - With latest change, Ketaki saw in her test ( 3 source nodes -> 5 dest nodes) 50-60% CPU on destination clusters, without views/indexes running on this cluster, no additional load on this system.
Hide
Permalink
Pavel Paulau added a comment -
It's still rather high in two cases:
-- initial bidir replication
-- initial unidir replication with 2 buckets and 2 replication streams

All other cases are fine.
Show
Pavel Paulau added a comment - It's still rather high in two cases: -- initial bidir replication -- initial unidir replication with 2 buckets and 2 replication streams All other cases are fine.
Hide
Permalink
Junyi Xie added a comment -
Pavel, with latest fix merged, do you still see the issue? Please try the build 1778 or later.
Show
Junyi Xie added a comment - Pavel, with latest fix merged, do you still see the issue? Please try the build 1778 or later.
Hide
Permalink
Pavel Paulau added a comment -
The simplest experiment:
-- 2 physical nodes as source, 2 physical nodes as destination. Both have ssd drives.
-- Load 10M x 2K items (lite DGM)
-- Start replication.

~9K new items/sec. Build 1779.
Show
Pavel Paulau added a comment - The simplest experiment: -- 2 physical nodes as source, 2 physical nodes as destination. Both have ssd drives. -- Load 10M x 2K items (lite DGM) -- Start replication. ~9K new items/sec. Build 1779.
Hide
Permalink
Junyi Xie added a comment -
Can you please test it with latest build? From your post on Yammer seems the CPU usage is OK at destination given high front-end workload.
Show
Junyi Xie added a comment - Can you please test it with latest build? From your post on Yammer seems the CPU usage is OK at destination given high front-end workload.
Hide
Permalink
Junyi Xie added a comment -
From Pavel's tests on Yammer, the CPU usage is 60-70%

https://www.yammer.com/couchbase.com/#/inbox/show?threadId=218452800
https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdE5FOGNuRzgzNFNvZnZQeXBqaFBDTmc#gid=4

Show
Junyi Xie added a comment - From Pavel's tests on Yammer, the CPU usage is 60-70% https://www.yammer.com/couchbase.com/#/inbox/show?threadId=218452800 https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdE5FOGNuRzgzNFNvZnZQeXBqaFBDTmc#gid=4
Hide
Permalink
Pavel Paulau added a comment -
Opening it again.

Frankly speaking 60-70% for initial unidirectional replication w/o front-end load, views and rebalance is too much.
Also take look at this chart:
https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdFVPRnBlUEpqSzZSOUg1TmRRcDl5RUE#gid=8

Sounds like regression. Even if this is .next candidate we should not close.
Show
Pavel Paulau added a comment - Opening it again. Frankly speaking 60-70% for initial unidirectional replication w/o front-end load, views and rebalance is too much. Also take look at this chart: https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdFVPRnBlUEpqSzZSOUg1TmRRcDl5RUE#gid=8 Sounds like regression. Even if this is .next candidate we should not close.
Hide
Permalink
Junyi Xie added a comment -
Is it a regression? What is the CPU usage you observed in previous build? I am under impression it is never less than 60-70%.

Note XDCR requires at least 3 ops per item at destination side. For a similar cluster without XDCR, if you apply a front-end workload which is 3 times the XDCR rate, what is the CPU usage? They should be comparable.
Show
Junyi Xie added a comment - Is it a regression? What is the CPU usage you observed in previous build? I am under impression it is never less than 60-70%. Note XDCR requires at least 3 ops per item at destination side. For a similar cluster without XDCR, if you apply a front-end workload which is 3 times the XDCR rate, what is the CPU usage? They should be comparable.
Hide
Permalink
Junyi Xie added a comment -
Just looked at the charts, actually the replication rate improves a from 4.5K to >7K (> 50% improvement) in bidir-XDCR , that explains why we see higher CPU usage.
Show
Junyi Xie added a comment - Just looked at the charts, actually the replication rate improves a from 4.5K to >7K (> 50% improvement) in bidir-XDCR , that explains why we see higher CPU usage.
Hide
Permalink
Junyi Xie added a comment -
Based on discussion with Pavel, and test results from Ketaki, it is ok to have current CPU usage and we will not try to further improve it in 2.0.
Show
Junyi Xie added a comment - Based on discussion with Pavel, and test results from Ketaki, it is ok to have current CPU usage and we will not try to further improve it in 2.0.
Hide
Permalink
Farshid Ghods added a comment -
this bug was deferred for 2.0 . reopening so that we will make some performance improvements beyond 2.0
Show
Farshid Ghods added a comment - this bug was deferred for 2.0 . reopening so that we will make some performance improvements beyond 2.0
Hide
Permalink
Junyi Xie added a comment -
Since we will address this issue post-.2.0, it should be a blocker to performance team.
Show
Junyi Xie added a comment - Since we will address this issue post-.2.0, it should be a blocker to performance team.

People

  • Assignee:
    Junyi Xie
    Reporter:
    Pavel Paulau
Vote (0)
Watch (0)

Dates

  • Created:
    Updated: