Couchbase Server

Growing queue on 2 of 11 nodes on a xdcr-cluster. Seeing constant replicator crash on xdcr-cluster

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Duplicate
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.1
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    2.0.1-153-rel
    1 bucket, bidirectional replication.

Description

System Test

Front end load @ 10k on both the clusters.
Bidirectional replication on-going on the cluster.

Deleted replication and recreated replication after a few hours, replication started from last checkpoints.

Most of the nodes, except 2 have successfully replicated data.

SAmple output from logs
debug.4:=========================CRASH REPORT=========================
debug.4- crasher:
debug.4- initial call: xdc_vbucket_rep:init/1
debug.4- pid: <0.4116.455>
debug.4- registered_name: []
debug.4- exception exit: {function_clause,
debug.4- [{couch_api_wrap,'-get_missing_revs/2-fun-1-',
debug.4- [400,
debug.4- [{"Cache-Control","must-revalidate"},
debug.4- {"Content-Length","50"},
debug.4- {"Content-Type","application/json"},
--


Adding more logs from both the clusters.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Ketaki Gangal added a comment -
Looks like it is not able to replicate to one particular node : Error replicating vbucket 229: {http_request_failed, "POST", "http://Administrator:*****@ec2-23-20-217-189.compute-
Show
Ketaki Gangal added a comment - Looks like it is not able to replicate to one particular node : Error replicating vbucket 229: {http_request_failed, "POST", " http://Administrator:*****@ec2-23-20-217-189.compute-
Hide
Permalink
Junyi Xie added a comment -
Is this one the same test as MB-7756? the source replicator keep crashing as expected because of the bad things happened on destination (MB-7756).

Any reason to file two bugs? Thanks.
Show
Junyi Xie added a comment - Is this one the same test as MB-7756 ? the source replicator keep crashing as expected because of the bad things happened on destination ( MB-7756 ). Any reason to file two bugs? Thanks.
Hide
Permalink
Junyi Xie added a comment -
Without complete log, the error msg you post is expected, because the destination is unreachable (probably the same as MB-7756)
Show
Junyi Xie added a comment - Without complete log, the error msg you post is expected, because the destination is unreachable (probably the same as MB-7756 )
Hide
Permalink
Ketaki Gangal added a comment -
Adding the logs.

This is a LInux cluster, it is differernt from 7756,
Show
Ketaki Gangal added a comment - Adding the logs. This is a LInux cluster, it is differernt from 7756,
Hide
Permalink
Junyi Xie added a comment -
Please also provide address if two clusters along with the log
Show
Junyi Xie added a comment - Please also provide address if two clusters along with the log
Hide
Permalink
Ketaki Gangal added a comment -
Source cluster ( growing queue) - http://ec2-23-20-140-198.compute-1.amazonaws.com:8091/index.html
Destination Cluster - ec2-23-20-108-25.compute-1.amazonaws.com:8091
Show
Ketaki Gangal added a comment - Source cluster ( growing queue) - http://ec2-23-20-140-198.compute-1.amazonaws.com:8091/index.html Destination Cluster - ec2-23-20-108-25.compute-1.amazonaws.com:8091
Hide
Permalink
Junyi Xie added a comment -
The bi-directional xdcr test has been running for >5 days, at this time, it looks healthy from xdcr perspective. No growing queue on both side, no unevenness observed.
See the screenshots from Ketaki's clusters



The only thing is there are meta data warnings at destination side.

[15:57:43] - Metadata overhead warning. Over 50% of RAM allocated to bucket "default" on node "ec2-23-20-108-25.compute-1.amazonaws.com" is taken up by keys and metadata.
Show
Junyi Xie added a comment - The bi-directional xdcr test has been running for >5 days, at this time, it looks healthy from xdcr perspective. No growing queue on both side, no unevenness observed. See the screenshots from Ketaki's clusters The only thing is there are meta data warnings at destination side. [15:57:43] - Metadata overhead warning. Over 50% of RAM allocated to bucket "default" on node "ec2-23-20-108-25.compute-1.amazonaws.com" is taken up by keys and metadata.
Hide
Permalink
Junyi Xie added a comment - - edited
Still do not see any skew or growing mutations to replicate as of now. The disk write queue, however is highly skewed.

I am not sure why it is a bug.
Show
Junyi Xie added a comment - - edited Still do not see any skew or growing mutations to replicate as of now. The disk write queue, however is highly skewed. I am not sure why it is a bug.
Hide
Permalink
Ketaki Gangal added a comment -
Hi Junyi,

The errors on replication were seen on the intial xdcr-start load phase. The system tests run the next phases w/ a smaller mutation rate and likely we will not see the same skew on this run again.

The XDCR queue is very small < 10K and < 1k on both the clusters. Likely one would not see unevenness on this state-cluster.

A lot of these XDCR errors are seen at particular points in replication and not continuously throughout the replication for most tests.

-Ketaki


Show
Ketaki Gangal added a comment - Hi Junyi, The errors on replication were seen on the intial xdcr-start load phase. The system tests run the next phases w/ a smaller mutation rate and likely we will not see the same skew on this run again. The XDCR queue is very small < 10K and < 1k on both the clusters. Likely one would not see unevenness on this state-cluster. A lot of these XDCR errors are seen at particular points in replication and not continuously throughout the replication for most tests. -Ketaki
Hide
Permalink
Jin Lim added a comment - - edited
Per bug scrubs, Ketaki to restart the test again and ping Junyi for when he should start monitor this issue.
Show
Jin Lim added a comment - - edited Per bug scrubs, Ketaki to restart the test again and ping Junyi for when he should start monitor this issue.
Hide
Permalink
Ketaki Gangal added a comment -
Duplicate of MB-7657
Show
Ketaki Gangal added a comment - Duplicate of MB-7657

People

  • Assignee:
    Ketaki Gangal
    Reporter:
    Ketaki Gangal
Vote (0)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: