Couchbase Server

XDCR load is high on idle buckets

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2.0-beta
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    single node cluster

Description

I create a bucket called "copy" with no password, and setup xdcr with a new cluster called "me" that lives at localhost:9000. Then I create a replication from default to me/copy and kick it off.

It works, a few minutes later, all my docs are in both buckets. But... the target bucket registers 2-3 thousand ops per second. These are GET ops. Not sure what is happening. Shows up even on buckets with less than five documents.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Peter Wansch added a comment -
Can you try to reproduce this one?
Show
Peter Wansch added a comment - Can you try to reproduce this one?
Hide
Permalink
Ketaki Gangal added a comment -
Seeing intermittently high cpu usage on idle bucket on destination cluster.

Steps:
------
Create a 2:2 node cluster.
Setup unidirectional replication from bucket1 to bucket1 of second cluster.
CPU stats are between 2-20% on destination, with some outliers at 50%.

Not seeing any high ops/sec on the cluster.
Show
Ketaki Gangal added a comment - Seeing intermittently high cpu usage on idle bucket on destination cluster. Steps: ------ Create a 2:2 node cluster. Setup unidirectional replication from bucket1 to bucket1 of second cluster. CPU stats are between 2-20% on destination, with some outliers at 50%. Not seeing any high ops/sec on the cluster.
Hide
Permalink
Ketaki Gangal added a comment -
Seeing very high(97-99%) CPU utilization on the destination cluster, which is relatively idle [All expected items are replicated. 6 items are in the disk-write queue and are not being drained to the disk].

From the top command seeing - CPU and Mem usage by beam.smp and memacached as very high.

Tasks: 94 total, 1 running, 93 sleeping, 0 stopped, 0 zombie
Cpu(s): 69.6%us, 20.8%sy, 0.0%ni, 1.8%id, 0.0%wa, 0.0%hi, 7.8%si, 0.0%st
Mem: 4060428k total, 3088308k used, 972120k free, 172236k buffers
Swap: 6430712k total, 0k used, 6430712k free, 1531416k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1890 couchbas 20 0 692m 377m 8476 S 134 9.5 222:15.59 beam.smp
 3272 couchbas 20 0 909m 762m 2964 S 62 19.2 113:48.32 memcached

Junyi - Could you reassign this to the right person?

                                                                           
 
Show
Ketaki Gangal added a comment - Seeing very high(97-99%) CPU utilization on the destination cluster, which is relatively idle [All expected items are replicated. 6 items are in the disk-write queue and are not being drained to the disk]. From the top command seeing - CPU and Mem usage by beam.smp and memacached as very high. Tasks: 94 total, 1 running, 93 sleeping, 0 stopped, 0 zombie Cpu(s): 69.6%us, 20.8%sy, 0.0%ni, 1.8%id, 0.0%wa, 0.0%hi, 7.8%si, 0.0%st Mem: 4060428k total, 3088308k used, 972120k free, 172236k buffers Swap: 6430712k total, 0k used, 6430712k free, 1531416k cached   PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND  1890 couchbas 20 0 692m 377m 8476 S 134 9.5 222:15.59 beam.smp  3272 couchbas 20 0 909m 762m 2964 S 62 19.2 113:48.32 memcached Junyi - Could you reassign this to the right person?                                                                              
Hide
Permalink
Peter Wansch added a comment -
Do we also see this behavior with compactor disabled?
Show
Peter Wansch added a comment - Do we also see this behavior with compactor disabled?
Hide
Permalink
Peter Wansch added a comment -
Damien's changes should make this bug go away.
Show
Peter Wansch added a comment - Damien's changes should make this bug go away.
Hide
Permalink
Junyi Xie added a comment -
Grab the latest code, my local test shows that after replicating around 1000 items, the CPU at destination is around 5-10%, which looks OK.

Show
Junyi Xie added a comment - Grab the latest code, my local test shows that after replicating around 1000 items, the CPU at destination is around 5-10%, which looks OK.

People

  • Assignee:
    Ketaki Gangal
    Reporter:
    J Chris Anderson
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: