Couchbase Server

Enabling use of index pausing during rebalance crashes set view group

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 2.0-beta
  • Fix Version/s: 2.0-beta
  • Component/s: ns_server, view-engine
  • Security Level: Public
  • Labels:
    None

Description

Running with my consistent views patches and enabled (initially disabled) index pausing:


# rlwrap ../send-diag-eval lh:9000 -q
ns_config:set(index_pausing_disabled, false).
ok
#

The following rebalance failed:

Rebalance exited with reason {{{{function_clause,
[{orddict,fetch,
[53,
[{96,9852},
{97,16395},
{98,16420},
{99,9837},
{100,9837},
{101,22988},
{102,16395},
{103,9852},
{104,9855},
{105,16385}]]},
{couch_set_view_group,
'-process_mark_as_unindexable/2-fun-0-',3},
{lists,foldl,3},
{couch_set_view_group,
process_mark_as_unindexable,2},
{couch_set_view_group,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
[<0.614.0>,
{mark_as_unindexable,"5"},
infinity]}},
{gen_server,call,
['capi_set_view_manager-default',
{set_vbucket_states,
[active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,active,active,active,active,
active,active,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,active,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,missing,missing,missing,missing,
missing,active,active,active,active,active,
active,active,active,active,active,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
replica,replica,replica,replica,replica,
missing],
[undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,paused,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined,
undefined,undefined,undefined,undefined]},
infinity]}},
{gen_server,call,
[{'janitor_agent-default','n_0@10.17.2.163'},
{if_rebalance,<0.2974.0>,
{update_vbucket_state,53,active,paused,
undefined}},
60000]}}


  1. File
    diag.xz
    08/Aug/12 7:46 PM
    810 kB
    Aleksey Kondratenko

Activity

Descending order - Click to sort in ascending order
Hide
Permalink
Aleksey Kondratenko added a comment -
Fixed now. thanks
Show
Aleksey Kondratenko added a comment - Fixed now. thanks
Hide
Permalink
Aleksey Kondratenko added a comment -
Indeed. I've started unpausing before removing vbucket and it seems to work
Show
Aleksey Kondratenko added a comment - Indeed. I've started unpausing before removing vbucket and it seems to work
Hide
Permalink
Aleksey Kondratenko added a comment -
Perhaps that's because I'm starting cleanup before unpausing vbucket.
Show
Aleksey Kondratenko added a comment - Perhaps that's because I'm starting cleanup before unpausing vbucket.
Hide
Permalink
Aleksey Kondratenko added a comment -
Thanks Filipe. I think there's similar crash now:


Event Module Code Server Node Time
Shutting down bucket "default" on 'n_0@10.17.2.163' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012
Bucket "default" loaded on node 'n_0@10.17.2.163' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012
Bucket "default" loaded on node 'n_0@10.17.2.163' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012
Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'n_0@10.17.2.163',
{'EXIT',
{{{bad_return_value,
{error,
<<"Intersection between requested cleanup list and current unindexable partitions: [64]">>}},
{gen_server,call,
['capi_set_view_manager-default',
{set_vbucket_states,
[active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,replica,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active,active,active,active,active,
active],
[undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined,undefined,undefined,
undefined]},
infinity]}},
{gen_server,call,
[{'janitor_agent-default',
'n_0@10.17.2.163'},
{if_rebalance,<0.3699.0>,
{update_vbucket_state,64,replica,
undefined,'n_1@10.17.2.163'}},
60000]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4},
{ns_vbucket_mover,
update_replication_post_move,3},
{ns_vbucket_mover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
Show
Aleksey Kondratenko added a comment - Thanks Filipe. I think there's similar crash now: Event Module Code Server Node Time Shutting down bucket "default" on ' n_0@10.17.2.163 ' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012 Bucket "default" loaded on node ' n_0@10.17.2.163 ' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012 Bucket "default" loaded on node ' n_0@10.17.2.163 ' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012 Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{' n_0@10.17.2.163 ', {'EXIT', {{{bad_return_value, {error, <<"Intersection between requested cleanup list and current unindexable partitions: [64]">>}}, {gen_server,call, ['capi_set_view_manager-default', {set_vbucket_states, [active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,replica, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active], [undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined]}, infinity]}}, {gen_server,call, [{'janitor_agent-default', ' n_0@10.17.2.163 '}, {if_rebalance,<0.3699.0>, {update_vbucket_state,64,replica, undefined,' n_1@10.17.2.163 '}}, 60000]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover, update_replication_post_move,3}, {ns_vbucket_mover,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]}
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-couchdb-preview #473 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/473/])
    MB-6165 Ignore index pausing of already paused partitions (Revision fc3a87e487f641ea80e56c4a48fd284442ee7388)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/src/couch_set_view_group.erl
Show
Thuan Nguyen added a comment - Integrated in github-couchdb-preview #473 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/473/ ])      MB-6165 Ignore index pausing of already paused partitions (Revision fc3a87e487f641ea80e56c4a48fd284442ee7388)      Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl
Hide
Permalink
Filipe Manana added a comment -
You asked for the same partition to be unindexable twice.
The second time it crashes because it's already marked as unindexable.

Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403
Show
Filipe Manana added a comment - You asked for the same partition to be unindexable twice. The second time it crashes because it's already marked as unindexable. Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403

People

  • Assignee:
    Filipe Manana
    Reporter:
    Aleksey Kondratenko
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: