Couchbase Server

Deleting bucket on a cluster gives error "Some nodes are still deleting bucket"

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: recent-builds-2.0
  • Fix Version/s: 2.0-beta-2
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Environment:
    build 1751 on Centos 64

Description

Steps to reproduce:

1. Create bucket on a single node
2. Load 50K items
3. Rebalance in 3 node
4. Wait for rebalance to complete and then delete the bucket.

Bucket deletion gives error(see screenshot):

"Some nodes are still deleting bucket"

Diagnostics are attached.
  1. GZip Archive
    10.3.3.104-8091-diag.txt.gz
    21/Sep/12 12:00 PM
    1.79 MB
    Deepkaran Salooja
  2. GZip Archive
    10.3.3.106-8091-diag.txt.gz
    21/Sep/12 12:00 PM
    2.84 MB
    Deepkaran Salooja
  3. GZip Archive
    10.3.3.107-8091-diag.txt.gz
    21/Sep/12 12:00 PM
    1.03 MB
    Deepkaran Salooja
  4. GZip Archive
    10.3.3.95-8091-diag.txt.gz
    21/Sep/12 12:00 PM
    2.52 MB
    Deepkaran Salooja
  1. BucketDelete.png
    130 kB
    21/Sep/12 12:00 PM

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Andrei Baranouski added a comment -
test:
http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/70/consoleFull

Show
Andrei Baranouski added a comment - test: http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/70/consoleFull
Hide
Permalink
Chiyoung Seo added a comment -
Mike,

Please take a look at the diag files to see if this issue was caused by the ep-engine side. Note that there was a regression (MB-6711) from our recent changes in ep-engine, which caused the flusher to keep failed continuously. I'm not sure if the root cause of this issue is from MB-6711.
Show
Chiyoung Seo added a comment - Mike, Please take a look at the diag files to see if this issue was caused by the ep-engine side. Note that there was a regression ( MB-6711 ) from our recent changes in ep-engine, which caused the flusher to keep failed continuously. I'm not sure if the root cause of this issue is from MB-6711 .
Hide
Permalink
Farshid Ghods added a comment -
Chiyoung
is MB-67-8 also related to the same change set ?
for today , to unblock QE I am locking down the ep-engine manifest to
https://github.com/couchbase/ep-engine/commit/f0352955cd4f2b60352d1ba6bac0d44446b8882f

we can unlock the manifest tomorrow to point to master/HEAD
Show
Farshid Ghods added a comment - Chiyoung is MB-67-8 also related to the same change set ? for today , to unblock QE I am locking down the ep-engine manifest to https://github.com/couchbase/ep-engine/commit/f0352955cd4f2b60352d1ba6bac0d44446b8882f we can unlock the manifest tomorrow to point to master/HEAD
Hide
Permalink
Jin Lim added a comment - - edited
Fashid, assuming you meant MB-6708 (not MB-67-8) it looks like the same issue. Somehow EP Engine gets stuck in infinite notifying loop of new header position of a given vbucket.
I am going to mark 6708 as a duplicate of MB6711 for now. BTW how this rebalance issue could have not revealed by running the testrunner's simple-test? We may want to ensure that such test case is part of the simple regression test. Thanks.
Show
Jin Lim added a comment - - edited Fashid, assuming you meant MB-6708 (not MB-67-8) it looks like the same issue. Somehow EP Engine gets stuck in infinite notifying loop of new header position of a given vbucket. I am going to mark 6708 as a duplicate of MB6711 for now. BTW how this rebalance issue could have not revealed by running the testrunner's simple-test? We may want to ensure that such test case is part of the simple regression test. Thanks.
Hide
Permalink
Jin Lim added a comment -
Sorry I meant to close 6708 instead of this. Thanks.
Show
Jin Lim added a comment - Sorry I meant to close 6708 instead of this. Thanks.
Hide
Permalink
Mike Wiederhold added a comment -
http://review.couchbase.org/#/c/21069/1

Note that creating and deleting a bucket very quickly might still cause this issue due to latencies between mccouch and ep-engine.
Show
Mike Wiederhold added a comment - http://review.couchbase.org/#/c/21069/1 Note that creating and deleting a bucket very quickly might still cause this issue due to latencies between mccouch and ep-engine.
Hide
Permalink
Deepkaran Salooja added a comment -
Verified with build 1820
Show
Deepkaran Salooja added a comment - Verified with build 1820

People

  • Assignee:
    Mike Wiederhold
    Reporter:
    Deepkaran Salooja
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: