Couchbase Server

Data loss during rebalance after service restart

Details

Description

* start development cluster of two nodes
* set up first node with default bucket
* join second node but don't rebalance yet
* load some data
* wait till it hits disk
* Ctrl-C and start this cluster back
* wait till warmup is done and observe that data is there
* run rebalance
* observe that after rebalance new node has _zero_ items. Those are lost.

It's not happening if we don't restart. And same happens if I just kill -9 memcached instead of restarting everything.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Yen added a comment -
Chiyoung, this showed up at the recent sprint demo... Can you analyze?
Show
Steve Yen added a comment - Chiyoung, this showed up at the recent sprint demo... Can you analyze?
Hide
Permalink
Keith Batten added a comment -
this is blocking warmup performance testing as well
Show
Keith Batten added a comment - this is blocking warmup performance testing as well
Hide
Permalink
Aleksey Kondratenko added a comment -
just to make it a bit more clear. The data is there after restart. It's just somehow not there after rebalance.
Show
Aleksey Kondratenko added a comment - just to make it a bit more clear. The data is there after restart. It's just somehow not there after rebalance.
Hide
Permalink
Keith Batten added a comment -
ok, i'll verify if i see this in my testing if i do a full service restart instead of just restarting memcached. if the data warms up properly with a service restart then i won't be blocked.
Show
Keith Batten added a comment - ok, i'll verify if i see this in my testing if i do a full service restart instead of just restarting memcached. if the data warms up properly with a service restart then i won't be blocked.
Hide
Permalink
Steve Yen added a comment -
Chiyoung, any update on this? Thx
Show
Steve Yen added a comment - Chiyoung, any update on this? Thx
Hide
Permalink
Chiyoung Seo added a comment -
This issue is a duplicate of http://www.couchbase.com/issues/browse/MB-4595.

It's a edge case that can only happen when all the nodes have the open checkpoint with id 1 and are restarted. I don't think this is a blocker, but will fix it for DP4 and 1.8.1.
Show
Chiyoung Seo added a comment - This issue is a duplicate of http://www.couchbase.com/issues/browse/MB-4595 . It's a edge case that can only happen when all the nodes have the open checkpoint with id 1 and are restarted. I don't think this is a blocker, but will fix it for DP4 and 1.8.1.

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Aleksey Kondratenko
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: