Couchbase Server

Don't delete database files when node leaves cluster.

Details

Description

A recent bug brought up a concern around the process of deleting the on-disk data files when a node is failed over or removed from the cluster.

I think it would be a better approach to preserve this data in the event that it is needed. Perhaps moving them out of the way and messaging to the user that they can clean them up manually when needed.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Aleksey Kondratenko added a comment -
Perry, you want files preserved for eternity or we can/should get rid of them when node is re-initialized or joins some cluster ?
Show
Aleksey Kondratenko added a comment - Perry, you want files preserved for eternity or we can/should get rid of them when node is re-initialized or joins some cluster ?
Hide
Permalink
Perry Krug added a comment -
I think the join cluster could be a fine situation to clean them up. What do you mean by re-initialized? This is really only needed as a fail-safe situation in case we need to recover the data from the node that was failed over or removed.
Show
Perry Krug added a comment - I think the join cluster could be a fine situation to clean them up. What do you mean by re-initialized? This is really only needed as a fail-safe situation in case we need to recover the data from the node that was failed over or removed.
Hide
Permalink
Aleksey Kondratenko added a comment -
that's when/if node is initialized via setup wizard as standalone cluster
Show
Aleksey Kondratenko added a comment - that's when/if node is initialized via setup wizard as standalone cluster
Hide
Permalink
Perry Krug added a comment -
Oh, yes...then that's a good time as well too. My only concern would be that the current code path re-initializes the node when it gets failed over or removed so we'd have to change that behavior.
Show
Perry Krug added a comment - Oh, yes...then that's a good time as well too. My only concern would be that the current code path re-initializes the node when it gets failed over or removed so we'd have to change that behavior.
Steve Yen made changes -
Field Original Value New Value
Fix Version/s 1.8.1 [ 10249 ]
Dipti Borkar made changes -
Assignee Aleksey Kondratenko [ alkondratenko ]
Dipti Borkar made changes -
Fix Version/s 1.8.1 [ 10295 ]
Fix Version/s 1.8.2 [ 10249 ]
Hide
Permalink
Steve Yen added a comment -
Aliaksey still needs final requirements on how this needs to behave.
Show
Steve Yen added a comment - Aliaksey still needs final requirements on how this needs to behave.
Hide
Permalink
Steve Yen added a comment -
Please document decision here.
Show
Steve Yen added a comment - Please document decision here.
Steve Yen made changes -
Assignee Aleksey Kondratenko [ alkondratenko ] Dipti Borkar [ dipti ]
Steve Yen made changes -
Priority Major [ 3 ] Blocker [ 1 ]
Hide
Permalink
Aleksey Kondratenko added a comment -
Aliaksey took this.

Decision was to don't touch files when node is ejected/failed over. But rid of them either in the middle of wizard or when rebalance back in starts.
Show
Aleksey Kondratenko added a comment - Aliaksey took this. Decision was to don't touch files when node is ejected/failed over. But rid of them either in the middle of wizard or when rebalance back in starts.
Aleksey Kondratenko made changes -
Assignee Dipti Borkar [ dipti ] Aliaksey Artamonau [ Aliaksey Artamonau ]
Hide
Permalink
Perry Krug added a comment -
Could we also add a task to notify the user when the files are about to be deleted (either in the wizard or when adding back / rebalancing)?
Show
Perry Krug added a comment - Could we also add a task to notify the user when the files are about to be deleted (either in the wizard or when adding back / rebalancing)?
Hide
Permalink
Aleksey Kondratenko added a comment -
how you want user to be notifed? We can use user-visible log message or we can have alert-like thing.
Show
Aleksey Kondratenko added a comment - how you want user to be notifed? We can use user-visible log message or we can have alert-like thing.
Hide
Permalink
Perry Krug added a comment -
I'd specifically like something that makes the user confirm that they understand that proceeding with whatever action will result in the deleting of files they may want to save. A log message after the fact doesn't give them the ability to recover that data.
Show
Perry Krug added a comment - I'd specifically like something that makes the user confirm that they understand that proceeding with whatever action will result in the deleting of files they may want to save. A log message after the fact doesn't give them the ability to recover that data.
Hide
Permalink
Steve Yen added a comment -
alk predicts this evening ETA
Show
Steve Yen added a comment - alk predicts this evening ETA
Hide
Permalink
Aleksey Kondratenko added a comment -
http://review.couchbase.org/14628
Show
Aleksey Kondratenko added a comment - http://review.couchbase.org/14628
Aleksey Kondratenko made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ns-server-2-0 #333 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/333/])
    MB-4564 Don't delete database files when node leaves cluster. (Revision ac961c9839396f64ef6c0ef56e5f298095a0da63)

     Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/ns_memcached.erl
* src/ns_cluster.erl
* src/menelaus_web_buckets.erl
* src/menelaus_web.erl
* src/ns_storage_conf.erl
* src/ns_rebalancer.erl
Show
Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #333 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/333/ ])      MB-4564 Don't delete database files when node leaves cluster. (Revision ac961c9839396f64ef6c0ef56e5f298095a0da63)      Result = SUCCESS Aliaksey Kandratsenka : Files : * src/ns_memcached.erl * src/ns_cluster.erl * src/menelaus_web_buckets.erl * src/menelaus_web.erl * src/ns_storage_conf.erl * src/ns_rebalancer.erl
Farshid Ghods made changes -
Summary Don't delete data files after failover or remove Don't delete database files when node leaves cluster.
Farshid Ghods made changes -
Labels customer 1.8.1-release-notes customer

People

  • Assignee:
    Aliaksey Artamonau
    Reporter:
    Perry Krug
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: