Couchbase Server

Error replicating vbucket XX: {badmatch, {error,timeout}} on a bidirectional replication setup.

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Cannot Reproduce
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.2
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    2.0.1-170-rc

Description


-Load 40M items on cluster1, 2M items on cluster2
- Setup a bidirectional replication between 2 clusters.
- No front end load on either cluster , after loading the intial data.

-After replicating 1.8M items , cluster2 replication shows errors w/ Badmatch timeout on serveral vbuckets.
- On cluster1, seeing replication errors as "Failed to grab remote bucket info from any of known nodes" after replicating 25M items.

Adding logs.

Adding screenshots from both clusters.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Ketaki Gangal added a comment -
Clusters :
http://ec2-23-20-140-198.compute-1.amazonaws.com:8091/index.html#sec=replications

http://ec2-23-22-53-131.compute-1.amazonaws.com:8091/index.html#sec=replications
Show
Ketaki Gangal added a comment - Clusters : http://ec2-23-20-140-198.compute-1.amazonaws.com:8091/index.html#sec=replications http://ec2-23-22-53-131.compute-1.amazonaws.com:8091/index.html#sec=replications
Hide
Permalink
Junyi Xie added a comment -
Looks like ns_server or ep_engine still have time_out issue we have hit many times before. If it is something expected and unavoidable, that means our system is unable to handle this type of XDCR workload.
Show
Junyi Xie added a comment - Looks like ns_server or ep_engine still have time_out issue we have hit many times before. If it is something expected and unavoidable, that means our system is unable to handle this type of XDCR workload.
Hide
Permalink
Ketaki Gangal added a comment -
https://s3.amazonaws.com/bugdb/jira/MB-7858/bug01.tar logs from the clusters.
Show
Ketaki Gangal added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7858/bug01.tar logs from the clusters.
Hide
Permalink
Junyi Xie added a comment -
Known timeout issue. Please assign to ns_server team.
Show
Junyi Xie added a comment - Known timeout issue. Please assign to ns_server team.
Hide
Permalink
Maria McDuff added a comment -
assigning to alk.
Show
Maria McDuff added a comment - assigning to alk.
Hide
Permalink
Maria McDuff added a comment -
ketaki, shld this be a blocker? assigning critical for now... pls update this bug and assign to alk for investigation.
also, is this sizing related, and does it recover?
Show
Maria McDuff added a comment - ketaki, shld this be a blocker? assigning critical for now... pls update this bug and assign to alk for investigation. also, is this sizing related, and does it recover?
Hide
Permalink
Maria McDuff added a comment -
assigning to alk k. issue is still there in 2.0.2
Show
Maria McDuff added a comment - assigning to alk k. issue is still there in 2.0.2
Hide
Permalink
Aleksey Kondratenko added a comment -
There is high chance that original issues is now addressed.

We need fresh logs in order to see if we're hitting same issue or not and if not what is new issue.
Show
Aleksey Kondratenko added a comment - There is high chance that original issues is now addressed. We need fresh logs in order to see if we're hitting same issue or not and if not what is new issue.
Hide
Permalink
Maria McDuff added a comment -
Alk, ok, we will get you fresh logs. thanks.
Show
Maria McDuff added a comment - Alk, ok, we will get you fresh logs. thanks.
Hide
Permalink
Ketaki Gangal added a comment -
Not able to reproduce this w/ 202-800 latest runs. Will reopen if seen again.
Show
Ketaki Gangal added a comment - Not able to reproduce this w/ 202-800 latest runs. Will reopen if seen again.

People

  • Assignee:
    Ketaki Gangal
    Reporter:
    Ketaki Gangal
Vote (0)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: