[MB-7554] Rebalance fails with "bad match wait_backfill_determination" error on a very small load

Mon, 4 Feb 2013 11:43:00 -0600

Load 1M items on a 4 node cluster.
Rebalance in 2 nodes.

Rebalance and Compaction start in parallel.

Rebalance is very slow in initial few minutes, catches up, but fails with a timeout exit,

The load/cluster is a very basic configiuration. This is a working on 2.0

** Reason for termination ==
** {unexpected_exit,
       {'EXIT',<0.896.2>,
           {{badmatch,
                [{'EXIT',
                     {timeout,
                         {gen_server,call,
                             [<20117.4759.0>,had_backfill,30000]}}}]},
            [{ns_single_vbucket_mover,
                 '-wait_backfill_determination/1-fun-1-',1}]}}}

[error_logger:error,2013-01-17T23:11:39.476,ns_1@10.176.169.6:error_logger<0.6.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_vbucket_mover:init/1
    pid: <0.24152.1>
    registered_name: []
    exception exit: {unexpected_exit,
                     {'EXIT',<0.896.2>,
                      {{badmatch,
                        [{'EXIT',
                          {timeout,
                           {gen_server,call,
                            [<20117.4759.0>,had_backfill,30000]}}}]},
                       [{ns_single_vbucket_mover,
                         '-wait_backfill_determination/1-fun-1-',1}]}}}
      in function gen_server:terminate/6
    ancestors: [<0.13807.1>]
    messages: [{backfill_done,
                      {'ns_1@10.176.169.6',1019,
                          ['ns_1@10.176.169.6','ns_1@10.169.54.218'],
                          ['ns_1@10.176.155.132','ns_1@10.168.94.60']}},
                  {move_done_new_style,
                      {'ns_1@10.176.169.6',1019,
                          ['ns_1@10.176.169.6','ns_1@10.169.54.218'],
                          ['ns_1@10.176.155.132','ns_1@10.168.94.60']}},
                  {'EXIT',<0.6884.2>,normal},
                  {backfill_done,
                      {'ns_1@10.169.54.218',678,
                          ['ns_1@10.169.54.218','ns_1@10.168.173.242'],
                          ['ns_1@10.168.94.60','ns_1@10.176.155.132']}},
                  {move_done_new_style,
                      {'ns_1@10.169.54.218',678,
                          ['ns_1@10.169.54.218','ns_1@10.168.173.242'],
                          ['ns_1@10.168.94.60','ns_1@10.176.155.132']}},
                  {'EXIT',<0.7095.2>,normal}]
    links: [<0.13807.1>,<0.24159.1>,<0.57.0>]
    dictionary: [{bucket_name,"default"},
                  {i_am_master_mover,true},
                  {child_processes,[<0.7095.2>,<0.6884.2>,<0.6760.2>,
                                    <0.3866.2>,<0.3858.2>,<0.862.2>,<0.855.2>,
                                    <0.852.2>,<0.787.2>,<0.26181.1>,
                                    <0.26131.1>,<0.26086.1>,<0.24174.1>,
                                    <0.24173.1>]}]
    trap_exit: true
    status: running
    heap_size: 28657
    stack_size: 24
    reductions: 1198089

Logs at

Couchbase

[MB-7554] Rebalance fails with "bad match wait_backfill_determination" error on a very small load