Couchbase Server

Seeing very frequent compaction on clusters, during xdcr

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Incomplete
  • Affects Version/s: 2.0-beta, 2.0
  • Fix Version/s: 2.0
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    2.0-1700
    xdcr, unidirectional replication.
    4g, 4 core machines
    1024 vbuckets

Description

There is almost continuous compaction happening on source/ destination clusters during xdcr replication.

-Loaded 2M data on source, this data is being replicated to the destination cluster.
-Compaction setting is set to 30% database fragmenation [default setting]

Not sure why it is triggered atleast 5-6 times and almost ongoing continuously on source/destination clusters.
Are there any other stats/memory that I should check for this?

- Every time compaction kicks in, stats on destination become choppy and replication becomes slower, due to resource contention.
- XDCR by iteself has very high CPU utilization ~90% on source.

Screenshots of cluster overview on source and destination.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
Alk,

was something changes recently in compaction manager ?
Show
Farshid Ghods added a comment - Alk, was something changes recently in compaction manager ?
Hide
Permalink
Ketaki Gangal added a comment -
How much data size/memory savings do we get with compaction? ie it should reduce significantly with every compaction, right?
Show
Ketaki Gangal added a comment - How much data size/memory savings do we get with compaction? ie it should reduce significantly with every compaction, right?
Hide
Permalink
Farshid Ghods added a comment -
depends on the type of workload you are running.
we dont save any space in memory , compaction compacts the data which is stored on disk
for instance if you load 1M items and delete half of them without compaction we will have 1.5 M items on the disk because we only do append only writes to disk and when compaction kicks in it deletes those items which are not valid anymore.
Show
Farshid Ghods added a comment - depends on the type of workload you are running. we dont save any space in memory , compaction compacts the data which is stored on disk for instance if you load 1M items and delete half of them without compaction we will have 1.5 M items on the disk because we only do append only writes to disk and when compaction kicks in it deletes those items which are not valid anymore.
Hide
Permalink
Aleksey Kondratenko added a comment -
There's not enough data.

For example compaction may be slower then usual, then given we compact vbucket after vbucket, some vbuckets at the end of compaction would already have lots of 'garbage' thus frequent compactions. For example.

I suggest observing all stats we have. Disk sizes & fragmentation stats particularly.

I can also try reproducing it myself.
Show
Aleksey Kondratenko added a comment - There's not enough data. For example compaction may be slower then usual, then given we compact vbucket after vbucket, some vbuckets at the end of compaction would already have lots of 'garbage' thus frequent compactions. For example. I suggest observing all stats we have. Disk sizes & fragmentation stats particularly. I can also try reproducing it myself.
Hide
Permalink
Chiyoung Seo added a comment -
This issue is not related to ep-engine as the compaction is totally controlled by the cluster manager and couchstore layer
Show
Chiyoung Seo added a comment - This issue is not related to ep-engine as the compaction is totally controlled by the cluster manager and couchstore layer
Hide
Permalink
Xiaoqin Ma added a comment -
Are there are lot of updates/create/delete going on? After each compaction finishes, what is the fraction ratio? If there are not much updates/create/delete items while compaction is running, normally when compaction is done, the fraction rate should become much smaller.
But with XDCR setup, if the fractions are caused by deletion mainly and those deletions haven't been propagated to destination yet, compaction will not really compact the data.
Show
Xiaoqin Ma added a comment - Are there are lot of updates/create/delete going on? After each compaction finishes, what is the fraction ratio? If there are not much updates/create/delete items while compaction is running, normally when compaction is done, the fraction rate should become much smaller. But with XDCR setup, if the fractions are caused by deletion mainly and those deletions haven't been propagated to destination yet, compaction will not really compact the data.

People

  • Assignee:
    Ketaki Gangal
    Reporter:
    Ketaki Gangal
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: