diag.xz
[MB-6165] Enabling use of index pausing during rebalance crashes set view group Created: 08/Aug/12 Updated: 09/Aug/12 Resolved: 09/Aug/12 |
|
| Status: | Resolved |
| Project: | Couchbase Server |
| Component/s: | ns_server, view-engine |
| Affects Version/s: | 2.0-beta |
| Fix Version/s: | 2.0-beta |
| Security Level: | Public |
| Type: | Bug | Priority: | Critical |
| Reporter: | Aleksey Kondratenko | Assignee: | Filipe Manana |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
diag.xz
|
| Description |
|
Running with my consistent views patches and enabled (initially disabled) index pausing:
# rlwrap ../send-diag-eval lh:9000 -q ns_config:set(index_pausing_disabled, false). ok # The following rebalance failed: Rebalance exited with reason {{{{function_clause, [{orddict,fetch, [53, [{96,9852}, {97,16395}, {98,16420}, {99,9837}, {100,9837}, {101,22988}, {102,16395}, {103,9852}, {104,9855}, {105,16385}]]}, {couch_set_view_group, '-process_mark_as_unindexable/2-fun-0-',3}, {lists,foldl,3}, {couch_set_view_group, process_mark_as_unindexable,2}, {couch_set_view_group,handle_call,3}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}, {gen_server,call, [<0.614.0>, {mark_as_unindexable,"5"}, infinity]}}, {gen_server,call, ['capi_set_view_manager-default', {set_vbucket_states, [active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,active,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,active,active,active,active,active, active,active,active,active,active,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing], [undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,paused,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined]}, infinity]}}, {gen_server,call, [{'janitor_agent-default','n_0@10.17.2.163'}, {if_rebalance,<0.2974.0>, {update_vbucket_state,53,active,paused, undefined}}, 60000]}} |
| Comments |
| Comment by Filipe Manana [ 09/Aug/12 ] |
|
You asked for the same partition to be unindexable twice.
The second time it crashes because it's already marked as unindexable. Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403 |
| Comment by Thuan Nguyen [ 09/Aug/12 ] |
|
Integrated in github-couchdb-preview #473 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/473/]) Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl |
| Comment by Aleksey Kondratenko [ 09/Aug/12 ] |
|
Thanks Filipe. I think there's similar crash now:
Event Module Code Server Node Time Shutting down bucket "default" on 'n_0@10.17.2.163' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012 Bucket "default" loaded on node 'n_0@10.17.2.163' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012 Bucket "default" loaded on node 'n_0@10.17.2.163' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012 Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{'n_0@10.17.2.163', {'EXIT', {{{bad_return_value, {error, <<"Intersection between requested cleanup list and current unindexable partitions: [64]">>}}, {gen_server,call, ['capi_set_view_manager-default', {set_vbucket_states, [active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,replica, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active], [undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined]}, infinity]}}, {gen_server,call, [{'janitor_agent-default', 'n_0@10.17.2.163'}, {if_rebalance,<0.3699.0>, {update_vbucket_state,64,replica, undefined,'n_1@10.17.2.163'}}, 60000]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} |
| Comment by Aleksey Kondratenko [ 09/Aug/12 ] |
| Perhaps that's because I'm starting cleanup before unpausing vbucket. |
| Comment by Aleksey Kondratenko [ 09/Aug/12 ] |
| Indeed. I've started unpausing before removing vbucket and it seems to work |
| Comment by Aleksey Kondratenko [ 09/Aug/12 ] |
| Fixed now. thanks |