[MB-6165] Enabling use of index pausing during rebalance crashes set view group Created: 08/Aug/12  Updated: 09/Aug/12  Resolved: 09/Aug/12

Status: Resolved
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: 2.0-beta
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aleksey Kondratenko Assignee: Filipe Manana
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File diag.xz    

 Description   
Running with my consistent views patches and enabled (initially disabled) index pausing:


# rlwrap ../send-diag-eval lh:9000 -q
ns_config:set(index_pausing_disabled, false).
ok
#

The following rebalance failed:

Rebalance exited with reason {{{{function_clause,
[{orddict,fetch,
[53,
[{96,9852},
{97,16395},
{98,16420},
{99,9837},
{100,9837},
{101,22988},
{102,16395},
{103,9852},
{104,9855},
{105,16385}]]},
{couch_set_view_group,
'-process_mark_as_unindexable/2-fun-0-',3},
{lists,foldl,3},
{couch_set_view_group,
process_mark_as_unindexable,2},
{couch_set_view_group,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
[<0.614.0>,
{mark_as_unindexable,"5"},
infinity]}},
{gen_server,call,
['capi_set_view_manager-default',
{set_vbucket_states,
[active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,active,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,active,active,active,active,active,
active,active,active,active,active,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
missing],
[undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,paused,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined]},
infinity]}},
{gen_server,call,
[{'janitor_agent-default','n_0@10.17.2.163'},
{if_rebalance,<0.2974.0>,
{update_vbucket_state,53,active,paused,
undefined}},
60000]}}




 Comments   
Comment by Filipe Manana [ 09/Aug/12 ]
You asked for the same partition to be unindexable twice.
The second time it crashes because it's already marked as unindexable.

Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403
Comment by Thuan Nguyen [ 09/Aug/12 ]
Integrated in github-couchdb-preview #473 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/473/])
    MB-6165 Ignore index pausing of already paused partitions (Revision fc3a87e487f641ea80e56c4a48fd284442ee7388)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/src/couch_set_view_group.erl
Comment by Aleksey Kondratenko [ 09/Aug/12 ]
Thanks Filipe. I think there's similar crash now:


Event Module Code Server Node Time
Shutting down bucket "default" on 'n_0@10.17.2.163' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012
Bucket "default" loaded on node 'n_0@10.17.2.163' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012
Bucket "default" loaded on node 'n_0@10.17.2.163' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012
Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'n_0@10.17.2.163',
{'EXIT',
{{{bad_return_value,
{error,
<<"Intersection between requested cleanup list and current unindexable partitions: [64]">>}},
{gen_server,call,
['capi_set_view_manager-default',
{set_vbucket_states,
[active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,replica,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active],
[undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined]},
infinity]}},
{gen_server,call,
[{'janitor_agent-default',
'n_0@10.17.2.163'},
{if_rebalance,<0.3699.0>,
{update_vbucket_state,64,replica,
undefined,'n_1@10.17.2.163'}},
60000]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4},
{ns_vbucket_mover,
update_replication_post_move,3},
{ns_vbucket_mover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
Comment by Aleksey Kondratenko [ 09/Aug/12 ]
Perhaps that's because I'm starting cleanup before unpausing vbucket.
Comment by Aleksey Kondratenko [ 09/Aug/12 ]
Indeed. I've started unpausing before removing vbucket and it seems to work
Comment by Aleksey Kondratenko [ 09/Aug/12 ]
Fixed now. thanks
Generated at Thu Apr 24 18:59:22 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.