rebalancing in a new 1.8 node if there are only 1 checkpoint open results in an imbalanced cluster because ep-engine in the source node does not backfill items from the open checkpoint

Thuan Nguyen added a comment - 26/Mar/12 12:53 AM Integrated in github-ep-engine-2-0 #230 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/230/ ])      MB-4595 Schedule backfill for a fresh client with empty data (Revision 303ab54372e422b122116a85f2f084071b1491ff)      Result = SUCCESS Chiyoung Seo : Files : * checkpoint.cc * tapconnection.cc * ep_testsuite.cc * checkpoint.hh

Chiyoung Seo added a comment - 25/Mar/12 9:19 PM http://review.couchbase.org/#change,14297

Chiyoung Seo added a comment - 27/Feb/12 9:59 PM As Mike is booked with QA-related tasks, I will work on this issue for 1.8.1 release.

Farshid Ghods added a comment - 14/Feb/12 4:42 PM then thats a different bug can you please open a new issue and mark it as blocker for 2.0 and 1.8.1 ( since now 1.8 is now based on master branch _

Aleksey Kondratenko added a comment - 14/Feb/12 4:39 PM I'm able to reproduce that deterministically on 2.0 with lots of items. With checkpoint ids greater then 0 and 1.

Chiyoung Seo added a comment - 29/Dec/11 4:29 PM This is a bug in checkpoint synchronization, but not a blocker for 1.8 release. This issue happens in the following scenario: 1) Set up the 1.7.x cluster and add a very small number of items (e.g., 100K items) into the cluster. At this time, each active vbucket has only one checkpoint with ID 1 2) Shut down the 1.7.x cluster and upgrade it to 1.8 and restart the cluster 3) During the warmup, each vbucket is loaded from disk and its open checkpoint id is updated from vbucket_state table in disk. In this case, each active vbucket open checkpoint still has an id 1, but won't have any items in the open checkpoint. 4) Add a brand-new 1.8 node into the cluster and rebalance. 5) Some active vbuckets are taken over to this new node. However, this does not require backfill operations on the existing nodes because they have the open checkpoint id 1 for all active vbuckets and the new node starts with the open checkpoint 1 as well. 6) After rebalance, the new node has 0 items on its active vbuckets. The above scenario is the corner case because we tested it with the 1.7.x cluster that has been running for a very short time that is less than a new checkpoint creation interval 10 minutes. Therefore, it is not likely to happen in our customers' clusters that have a long running 1.7.x membase and large open checkpoint ids for active vbuckets.

Farshid Ghods added a comment - 29/Dec/11 12:00 AM this bug is not a regression and exist in current installations but only happen if user restarts a node and attempts to add and rebalance new nodes when existing nodes only have 1 checkpoint ( cluster has been running less than an hour or there are less than 500k items in the cluster )

Karan Kumar added a comment - 28/Dec/11 9:03 PM Assigning this to Chiyoung. This turned out to be a bug in ep-engine with checkpoint management. We want to make sure that before rebalancing in new nodes into the cluster, the old nodes have checkpoint greater than 1. Verified that the issue does not occur when checkpoint id is greater than 1.

Karan Kumar added a comment - 28/Dec/11 4:01 PM Also, Before rebalance had total 54K keys. (on 4 nodes) After rebalance I see only 36K in total . (in the screenshot)

Karan Kumar added a comment - 28/Dec/11 3:46 PM On a separate cluster was able to reproduce this. Attaching the screenshot after rebalance.