Couchbase Server

active/replica mismatch when rebalancing out a node with 100 items , replica=2 where some of the items are deleted ( seems like deletion is not replicated )

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
    None

Description

build-1395

case to reproduce:
./testrunner -i FILE.ini -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=1,replicas=2,items=100,doc_ops=create;update;delete

steps for the case, for instance:
1.created cluster with 7 nodes and replica=2, create default bucket
2.upload 100 docs
  Active / Replica = 100/200 - OK
3.rebalance out 1 node and during rebalancing perform doc's operation:
   a) update 49 docs
        Active / Replica = 100/200 - OK
   b) create 49 docs
       Active / Replica = 149/298 - OK
   c) delete 50 items


result:
     Active / Replica = 99/298 - value for replica was not changed!

This is a regression, many rebalance tests fell

also, the similar steps with screenshots can be found http://www.couchbase.com/issues/browse/MB-5754




  1. Text File
    node1.txt
    02/Jul/12 12:31 PM
    11 kB
    Farshid Ghods
  2. Text File
    node1tap.txt
    02/Jul/12 12:31 PM
    9 kB
    Farshid Ghods
  3. Text File
    node2.txt
    02/Jul/12 12:31 PM
    11 kB
    Farshid Ghods
  4. Text File
    node2.txt
    02/Jul/12 12:31 PM
    11 kB
    Farshid Ghods
  5. Text File
    node3.txt
    02/Jul/12 12:31 PM
    11 kB
    Farshid Ghods
  6. Text File
    node3tap.txt
    02/Jul/12 12:31 PM
    9 kB
    Farshid Ghods
  1. replica_vBucket.png
    103 kB
    01/Jul/12 6:11 PM
  2. Screen Shot 2012-07-02 at 10.31.32 AM.png
    9 kB
    02/Jul/12 12:31 PM

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
Andrei is going to post cbstats all and tap from all nodes in the cluster and verify whether this data loss happens when adding nodes as well as removing nodes
Show
Farshid Ghods added a comment - Andrei is going to post cbstats all and tap from all nodes in the cluster and verify whether this data loss happens when adding nodes as well as removing nodes
Hide
Permalink
Andrei Baranouski added a comment -
doesn't matter is rebalance or not, manual steps(1 doc as example) with screenshots can be found http://www.couchbase.com/issues/browse/MB-5754 :
add 1 doc, remove 1 doc -> total replica is still equal 2( for replica =2)

or the same test with min steps:
1.created cluster with 7 nodes and replica=2, create default bucket
2.upload 100 docs
  Active / Replica = 100/200 - OK
3.delete 50 docs
result:
     Active / Replica = 50/200 - value for replica was not changed!

Show
Andrei Baranouski added a comment - doesn't matter is rebalance or not, manual steps(1 doc as example) with screenshots can be found http://www.couchbase.com/issues/browse/MB-5754 : add 1 doc, remove 1 doc -> total replica is still equal 2( for replica =2) or the same test with min steps: 1.created cluster with 7 nodes and replica=2, create default bucket 2.upload 100 docs   Active / Replica = 100/200 - OK 3.delete 50 docs result:      Active / Replica = 50/200 - value for replica was not changed!
Hide
Permalink
Farshid Ghods added a comment -
Hi Andrei,

can you please post cbstats all and diags from the nodes ?
Show
Farshid Ghods added a comment - Hi Andrei, can you please post cbstats all and diags from the nodes ?
Hide
Permalink
Farshid Ghods added a comment - - edited
to reproduce this you can run this command against cluster run with 4 nodes (

./testrunner -i b/resources/dev-4-nodes.ini -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=1,delete-ratio=1,expiry-ratio=0,keys-count=1000

the test passes when delete-ratio=0
Show
Farshid Ghods added a comment - - edited to reproduce this you can run this command against cluster run with 4 nodes ( ./testrunner -i b/resources/dev-4-nodes.ini -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=1,delete-ratio=1,expiry-ratio=0,keys-count=1000 the test passes when delete-ratio=0
Hide
Permalink
Farshid Ghods added a comment -
attached tap and all stats from 3 node cluster where the current items does not match replica items after deletion of some items
Show
Farshid Ghods added a comment - attached tap and all stats from 3 node cluster where the current items does not match replica items after deletion of some items
Hide
Permalink
Chiyoung Seo added a comment -
http://review.couchbase.org/#change,17830
Show
Chiyoung Seo added a comment - http://review.couchbase.org/#change,17830

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Andrei Baranouski
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: