[MB-6415] Failure to open view group when rebalance is ongoing Created: 24/Aug/12  Updated: 11/Sep/12  Resolved: 28/Aug/12

Status: Closed
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.0-beta
Fix Version/s: 2.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Karan Kumar (Inactive) Assignee: Filipe Manana
Resolution: Fixed Votes: 0
Labels: regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 2.0.0-1633-rel

Attachments: GZip Archive 2b13dc62-cb9f-41ef-883a-380ececa15ab-10.3.3.111-diag.txt.gz     GZip Archive 2b13dc62-cb9f-41ef-883a-380ececa15ab-10.3.3.116-diag.txt.gz     GZip Archive 2b13dc62-cb9f-41ef-883a-380ececa15ab-10.3.3.119-diag.txt.gz     GZip Archive 2b13dc62-cb9f-41ef-883a-380ececa15ab-10.3.3.120-diag.txt.gz     GZip Archive 2b13dc62-cb9f-41ef-883a-380ececa15ab-10.3.3.123-diag.txt.gz     GZip Archive 6cdcf9ae-89bb-41c1-b2f8-38143133a575-10.3.3.111-diag.gz     GZip Archive 6cdcf9ae-89bb-41c1-b2f8-38143133a575-10.3.3.116-diag.gz     GZip Archive 6cdcf9ae-89bb-41c1-b2f8-38143133a575-10.3.3.119-diag.gz     GZip Archive 6cdcf9ae-89bb-41c1-b2f8-38143133a575-10.3.3.120-diag.gz     GZip Archive 6cdcf9ae-89bb-41c1-b2f8-38143133a575-10.3.3.123-diag.gz    

 Description   
There are many other bugs around the same area, but they seem to be timeout related.

Not completely sure, if this is same as them. Filing for being safe.
http://www.couchbase.com/issues/browse/MB-6284
http://www.couchbase.com/issues/browse/MB-5978


[rebalance:debug,2012-08-24T15:35:58.601,ns_1@10.3.3.123:<0.151.24>:janitor_agent:bulk_set_vbucket_state:374]bulk vbucket state change failed for:
[{'ns_1@10.3.3.120',
     {'EXIT',
         {killed,
             {gen_server,call,
                 [{'janitor_agent-default','ns_1@10.3.3.120'},
                  {if_rebalance,<0.32709.23>,
                      {update_vbucket_state,70,replica,undefined,undefined}},
                  infinity]}}}}]



[error_logger:error,2012-08-24T15:35:58.829,ns_1@10.3.3.123:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_vbucket_mover:init/1
    pid: <0.32709.23>
    registered_name: []
    exception exit: {{bulk_set_vbucket_state_failed,
                      [{'ns_1@10.3.3.123',
                        {'EXIT',
                         {killed,
                          {gen_server,call,
                           [{'janitor_agent-default','ns_1@10.3.3.123'},
                            {if_rebalance,<0.32709.23>,
                             {update_vbucket_state,0,replica,undefined,
                              'ns_1@10.3.3.120'}},
                            infinity]}}}}]},
                     [{janitor_agent,bulk_set_vbucket_state,4},
                      {ns_vbucket_mover,update_replication_post_move,3},
                      {ns_vbucket_mover,handle_info,2},
                      {gen_server,handle_msg,5},
                      {proc_lib,init_p_do_apply,3}]}
      in function gen_server:terminate/6
    ancestors: [<0.32629.23>]
    messages: [{'EXIT',<0.32716.23>,normal},
                  {'EXIT',<0.151.24>,
                   {bulk_set_vbucket_state_failed,
                    [{'ns_1@10.3.3.120',
                      {'EXIT',
                       {killed,
                        {gen_server,call,
                         [{'janitor_agent-default','ns_1@10.3.3.120'},
                          {if_rebalance,<0.32709.23>,
                           {update_vbucket_state,70,replica,undefined,
                            undefined}},
                          infinity]}}}}]}},
                  {'EXIT',<0.98.24>,
                   {killed,
                    {gen_server,call,
                     [{'janitor_agent-default','ns_1@10.3.3.120'},
                      {if_rebalance,<0.32709.23>,
                       {update_vbucket_state,53,active,undefined,undefined}},
                      infinity]}}}]
    links: [<0.32629.23>,<0.32714.23>,<0.62.0>]
    dictionary: [{bucket_name,"default"},
                  {i_am_master_mover,true},
                  {child_processes,[<0.151.24>,<0.98.24>,<0.32718.23>,
                                    <0.32717.23>,<0.32716.23>]}]


 Comments   
Comment by Farshid Ghods [ 24/Aug/12 ]
also fails in make simple-test
Comment by Aleksey Kondratenko [ 24/Aug/12 ]
Here's what I'm seeing:


[views:info,2012-08-24T15:35:58.218,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:353]
couch_set_view:set_partition_states([<<"default">>,<<"_design/dev_ddoc21">>,
                                     [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
                                      16,17,18,19,20,21,22,23,24,25,26,27,28,
                                      29,30,31,32,33,34,35,36,37,38,39,40,41,
                                      42,43,44,45,46,47,48,49,50,51,205,206,
                                      207,208,209,210,211,212,213,214,215,216,
                                      217,218,219,220,221,222,223,224,225,226,
                                      227,228,229,230,231,232,233,234,235,236,
                                      237,238,239,240,241,242,243,244,245,246,
                                      247,248,249,250,251,252,253,254,255,256,
                                      257,258,259,260,261,262,263,264,265,266,
                                      267,268,269,270,271,272,273,274,275,276,
                                      277,278,279,280,281,282,283,284,285,286,
                                      287,288,289,290,291,292,293,294,295,296,
                                      297,298,299,300,301,302,303,304,305,306,
                                      307,308,309,310,311,312,313,314,315,316,
                                      317,318,319,320,321,322,323,324,325,326,
                                      327,328,329,330,331,332,333,334,335,336,
                                      337,338,339,340,341,342,343,344,345,346,
                                      347,348,349,350,351,352,353,354,355,356,
                                      357,358,359,360,361,362,363,364,365,366,
                                      367,368,369,370,371,372,373,374,375,376,
                                      377,378,379,380,381,382,383,384,385,386,
                                      387,388,389,390,391,392,393,394,395,396,
                                      397,398,399,400,401,402,403,404,405,406,
                                      407,408,409,410,411,412,413,414,415,416,
                                      417,418,419,420,421,422,423,424,425,426,
                                      427,428,429,430,431,432,433,434,435,436,
                                      437,438,439,440,441,442,443,461,462,463,
                                      464,465,466,467,468,469,470,471,472,473,
                                      474,475,476,477,478,479,480,481,482,483,
                                      484,485,486,487,488,489,490,491,492,493,
                                      494,495,496,497,498,499,500,501,502,503,
                                      504,505,506,507,508,509,510,511],
                                     [],
                                     [52,53,54,55,56,57,58,59,60,61,62,63,64,
                                      65,66,67,68,69,70,71,72,73,74,75,76,77,
                                      78,79,80,81,82,83,84,85,86,87,88,89,90,
                                      91,92,93,94,95,96,97,98,99,100,101,102,
                                      103,104,105,106,107,108,109,110,111,112,
                                      113,114,115,116,117,118,119,120,121,122,
                                      123,124,125,126,127,128,129,130,131,132,
                                      133,134,135,136,137,138,139,140,141,142,
                                      143,144,145,146,147,148,149,150,151,152,
                                      153,154,155,156,157,158,159,160,161,162,
                                      163,164,165,166,167,168,169,170,171,172,
                                      173,174,175,176,177,178,179,180,181,182,
                                      183,184,185,186,187,188,189,190,191,192,
                                      193,194,195,196,197,198,199,200,201,202,
                                      203,204,444,445,446,447,448,449,450,451,
                                      452,453,454,455,456,457,458,459,460,512,
                                      513,514,515,516,517,518,519,520,521,522,
                                      523,524,525,526,527,528,529,530,531,532,
                                      533,534,535,536,537,538,539,540,541,542,
                                      543,544,545,546,547,548,549,550,551,552,
                                      553,554,555,556,557,558,559,560,561,562,
                                      563,564,565,566,567,568,569,570,571,572,
                                      573,574,575,576,577,578,579,580,581,582,
                                      583,584,585,586,587,588,589,590,591,592,
                                      593,594,595,596,597,598,599,600,601,602,
                                      603,604,605,606,607,608,609,610,611,612,
                                      613,614,615,616,617,618,619,620,621,622,
                                      623,624,625,626,627,628,629,630,631,632,
                                      633,634,635,636,637,638,639,640,641,642,
                                      643,644,645,646,647,648,649,650,651,652,
                                      653,654,655,656,657,658,659,660,661,662,
                                      663,664,665,666,667,668,669,670,671,672,
                                      673,674,675,676,677,678,679,680,681,682,
                                      683,684,685,686,687,688,689,690,691,692,
                                      693,694,695,696,697,698,699,700,701,702,
                                      703,704,705,706,707,708,709,710,711,712,
                                      713,714,715,716,717,718,719,720,721,722,
                                      723,724,725,726,727,728,729,730,731,732,
                                      733,734,735,736,737,738,739,740,741,742,
                                      743,744,745,746,747,748,749,750,751,752,
                                      753,754,755,756,757,758,759,760,761,762,
                                      763,764,765,766,767,768,769,770,771,772,
                                      773,774,775,776,777,778,779,780,781,782,
                                      783,784,785,786,787,788,789,790,791,792,
                                      793,794,795,796,797,798,799,800,801,802,
                                      803,804,805,806,807,808,809,810,811,812,
                                      813,814,815,816,817,818,819,820,821,822,
                                      823,824,825,826,827,828,829,830,831,832,
                                      833,834,835,836,837,838,839,840,841,842,
                                      843,844,845,846,847,848,849,850,851,852,
                                      853,854,855,856,857,858,859,860,861,862,
                                      863,864,865,866,867,868,869,870,871,872,
                                      873,874,875,876,877,878,879,880,881,882,
                                      883,884,885,886,887,888,889,890,891,892,
                                      893,894,895,896,897,898,899,900,901,902,
                                      903,904,905,906,907,908,909,910,911,912,
                                      913,914,915,916,917,918,919,920,921,922,
                                      923,924,925,926,927,928,929,930,931,932,
                                      933,934,935,936,937,938,939,940,941,942,
                                      943,944,945,946,947,948,949,950,951,952,
                                      953,954,955,956,957,958,959,960,961,962,
                                      963,964,965,966,967,968,969,970,971,972,
                                      973,974,975,976,977,978,979,980,981,982,
                                      983,984,985,986,987,988,989,990,991,992,
                                      993,994,995,996,997,998,999,1000,1001,
                                      1002,1003,1004,1005,1006,1007,1008,1009,
                                      1010,1011,1012,1013,1014,1015,1016,1017,
                                      1018,1019,1020,1021,1022,1023]]) returned ok in 30ms
[views:info,2012-08-24T15:35:58.221,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:357]
Calling couch_set_view:add_replica_partitions([<<"default">>,
                                               <<"_design/dev_ddoc21">>,
                                               [53,54,55,56,57,58,59,60,61,62,
                                                63,64,65,66,67,68,69,70,71,72,
                                                73,74,75,76,77,78,79,80,81,82,
                                                83,84,85,86,87,88,89,90,91,92,
                                                93,94,95,96,97,98,99,100,101,
                                                102,103,104,105,106,107,108,
                                                109,110,111,112,113,114,115,
                                                116,117,118,119,120,121,122,
                                                123,124,125,126,127,128,129,
                                                130,131,132,133,134,135,136,
                                                137,138,139,140,141,142,143,
                                                144,145,146,147,148,149,150,
                                                151,152,153,154,155,156,157,
                                                158,159,160,161,162,163,164,
                                                165,166,167,168,169,170,171,
                                                172,173,174,175,176,177,178,
                                                179,180,181,182,183,184,185,
                                                186,187,188,189,190,191,192,
                                                193,194,195,196,197,198,199,
                                                200,201,202,203,204,444,445,
                                                446,447,448,449,450,451,452,
                                                453,454,455,456,457,458,459,
                                                460,512,546,547,548,549,550,
                                                551,552,553,554,555,556,557,
                                                558,559,560,561,562,563,564,
                                                598,599,600,601,602,603,604,
                                                605,606,607,608,609,610,611,
                                                612,613,614,666,667,668,669,
                                                670,671,672,673,674,675,676,
                                                677,678,679,680,681,682,683,
                                                684,685,686,687,688,689,690,
                                                691,692,693,694,695,696,697,
                                                698,699,700,701,702,703,704,
                                                705,706,707,708,709,710,711,
                                                712,713,714,715,716,717,718,
                                                719,720,721,722,723,724,725,
                                                726,727,728,729,730,731,732,
                                                768,871,872,873,874,875,876,
                                                877,878,879,880,881,882,883,
                                                884,885,886,887,888,889,890,
                                                891,892,893,894,895,896,897,
                                                898,899,900,901,902,903,904,
                                                905,906,907,908,909,910,911,
                                                912,913,914,915,916,917,918,
                                                919,920,921,922,923,924,925,
                                                926,927,928,929,930,931,932,
                                                933,934,935,936,937]])
[couchdb:info,2012-08-24T15:35:58.222,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, defined new replica partitions: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
New full set of replica partitions is: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]

[views:info,2012-08-24T15:35:58.246,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:358]
couch_set_view:add_replica_partitions([<<"default">>,<<"_design/dev_ddoc21">>,
                                       [53,54,55,56,57,58,59,60,61,62,63,64,
                                        65,66,67,68,69,70,71,72,73,74,75,76,
                                        77,78,79,80,81,82,83,84,85,86,87,88,
                                        89,90,91,92,93,94,95,96,97,98,99,100,
                                        101,102,103,104,105,106,107,108,109,
                                        110,111,112,113,114,115,116,117,118,
                                        119,120,121,122,123,124,125,126,127,
                                        128,129,130,131,132,133,134,135,136,
                                        137,138,139,140,141,142,143,144,145,
                                        146,147,148,149,150,151,152,153,154,
                                        155,156,157,158,159,160,161,162,163,
                                        164,165,166,167,168,169,170,171,172,
                                        173,174,175,176,177,178,179,180,181,
                                        182,183,184,185,186,187,188,189,190,
                                        191,192,193,194,195,196,197,198,199,
                                        200,201,202,203,204,444,445,446,447,
                                        448,449,450,451,452,453,454,455,456,
                                        457,458,459,460,512,546,547,548,549,
                                        550,551,552,553,554,555,556,557,558,
                                        559,560,561,562,563,564,598,599,600,
                                        601,602,603,604,605,606,607,608,609,
                                        610,611,612,613,614,666,667,668,669,
                                        670,671,672,673,674,675,676,677,678,
                                        679,680,681,682,683,684,685,686,687,
                                        688,689,690,691,692,693,694,695,696,
                                        697,698,699,700,701,702,703,704,705,
                                        706,707,708,709,710,711,712,713,714,
                                        715,716,717,718,719,720,721,722,723,
                                        724,725,726,727,728,729,730,731,732,
                                        768,871,872,873,874,875,876,877,878,
                                        879,880,881,882,883,884,885,886,887,
                                        888,889,890,891,892,893,894,895,896,
                                        897,898,899,900,901,902,903,904,905,
                                        906,907,908,909,910,911,912,913,914,
                                        915,916,917,918,919,920,921,922,923,
                                        924,925,926,927,928,929,930,931,932,
                                        933,934,935,936,937]]) returned ok in 23ms
[views:info,2012-08-24T15:35:58.248,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:359]
Calling couch_set_view:remove_replica_partitions([<<"default">>,
                                                  <<"_design/dev_ddoc21">>,
                                                  [52,513,514,515,516,517,518,
                                                   519,520,521,522,523,524,
                                                   525,526,527,528,529,530,
                                                   531,532,533,534,535,536,
                                                   537,538,539,540,541,542,
                                                   543,544,545,565,566,567,
                                                   568,569,570,571,572,573,
                                                   574,575,576,577,578,579,
                                                   580,581,582,583,584,585,
                                                   586,587,588,589,590,591,
                                                   592,593,594,595,596,597,
                                                   615,616,617,618,619,620,
                                                   621,622,623,624,625,626,
                                                   627,628,629,630,631,632,
                                                   633,634,635,636,637,638,
                                                   639,640,641,642,643,644,
                                                   645,646,647,648,649,650,
                                                   651,652,653,654,655,656,
                                                   657,658,659,660,661,662,
                                                   663,664,665,733,734,735,
                                                   736,737,738,739,740,741,
                                                   742,743,744,745,746,747,
                                                   748,749,750,751,752,753,
                                                   754,755,756,757,758,759,
                                                   760,761,762,763,764,765,
                                                   766,767,769,770,771,772,
                                                   773,774,775,776,777,778,
                                                   779,780,781,782,783,784,
                                                   785,786,787,788,789,790,
                                                   791,792,793,794,795,796,
                                                   797,798,799,800,801,802,
                                                   803,804,805,806,807,808,
                                                   809,810,811,812,813,814,
                                                   815,816,817,818,819,820,
                                                   821,822,823,824,825,826,
                                                   827,828,829,830,831,832,
                                                   833,834,835,836,837,838,
                                                   839,840,841,842,843,844,
                                                   845,846,847,848,849,850,
                                                   851,852,853,854,855,856,
                                                   857,858,859,860,861,862,
                                                   863,864,865,866,867,868,
                                                   869,870,938,939,940,941,
                                                   942,943,944,945,946,947,
                                                   948,949,950,951,952,953,
                                                   954,955,956,957,958,959,
                                                   960,961,962,963,964,965,
                                                   966,967,968,969,970,971,
                                                   972,973,974,975,976,977,
                                                   978,979,980,981,982,983,
                                                   984,985,986,987,988,989,
                                                   990,991,992,993,994,995,
                                                   996,997,998,999,1000,1001,
                                                   1002,1003,1004,1005,1006,
                                                   1007,1008,1009,1010,1011,
                                                   1012,1013,1014,1015,1016,
                                                   1017,1018,1019,1020,1021,
                                                   1022,1023]])
[couchdb:info,2012-08-24T15:35:58.269,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, partition states updated
active partitions before: []
active partitions after: []
passive partitions before: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
cleanup partitions before: []
cleanup partitions after: [52]
unindexable partitions: []
replica partitions before: []
replica partitions after: []
replicas on transfer before: []
replicas on transfer after: []
pending transition before:
  active: []
  passive: []
pending transition after:
  active: []
  passive: []

[couchdb:info,2012-08-24T15:35:58.270,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Started cleanup process <0.30364.2> for set view `default`, replica group `_design/dev_ddoc21`
[couchdb:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, marked the following replica partitions for removal: [52,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]
[couchdb:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Cleanup finished for set view `default`, replica group `_design/dev_ddoc21`
Removed 0 values from the index in 0.000 seconds
active partitions before: []
active partitions after: []
passive partitions before: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
cleanup partitions before: [52]
cleanup partitions after: []

[views:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:360]
couch_set_view:remove_replica_partitions([<<"default">>,
                                          <<"_design/dev_ddoc21">>,
                                          [52,513,514,515,516,517,518,519,520,
                                           521,522,523,524,525,526,527,528,
                                           529,530,531,532,533,534,535,536,
                                           537,538,539,540,541,542,543,544,
                                           545,565,566,567,568,569,570,571,
                                           572,573,574,575,576,577,578,579,
                                           580,581,582,583,584,585,586,587,
                                           588,589,590,591,592,593,594,595,
                                           596,597,615,616,617,618,619,620,
                                           621,622,623,624,625,626,627,628,
                                           629,630,631,632,633,634,635,636,
                                           637,638,639,640,641,642,643,644,
                                           645,646,647,648,649,650,651,652,
                                           653,654,655,656,657,658,659,660,
                                           661,662,663,664,665,733,734,735,
                                           736,737,738,739,740,741,742,743,
                                           744,745,746,747,748,749,750,751,
                                           752,753,754,755,756,757,758,759,
                                           760,761,762,763,764,765,766,767,
                                           769,770,771,772,773,774,775,776,
                                           777,778,779,780,781,782,783,784,
                                           785,786,787,788,789,790,791,792,
                                           793,794,795,796,797,798,799,800,
                                           801,802,803,804,805,806,807,808,
                                           809,810,811,812,813,814,815,816,
                                           817,818,819,820,821,822,823,824,
                                           825,826,827,828,829,830,831,832,
                                           833,834,835,836,837,838,839,840,
                                           841,842,843,844,845,846,847,848,
                                           849,850,851,852,853,854,855,856,
                                           857,858,859,860,861,862,863,864,
                                           865,866,867,868,869,870,938,939,
                                           940,941,942,943,944,945,946,947,
                                           948,949,950,951,952,953,954,955,
                                           956,957,958,959,960,961,962,963,
                                           964,965,966,967,968,969,970,971,
                                           972,973,974,975,976,977,978,979,
                                           980,981,982,983,984,985,986,987,
                                           988,989,990,991,992,993,994,995,
                                           996,997,998,999,1000,1001,1002,
                                           1003,1004,1005,1006,1007,1008,1009,
                                           1010,1011,1012,1013,1014,1015,1016,
                                           1017,1018,1019,1020,1021,1022,
                                           1023]]) returned ok in 21ms
[ns_server:debug,2012-08-24T15:35:58.273,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':cb_generic_replication_srv:handle_cast:131]Writing replicated ddoc {doc,<<"_design/dev_ddoc21">>,
                             {2,<<67,158,20,199>>},
                             <<"{\"views\":{\"views0\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"},\"views1\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"}}}">>,
                             0,false,[]}
[ns_server:info,2012-08-24T15:35:58.273,ns_1@10.3.3.116:<0.31061.1>:mc_connection:do_delete_vbucket:123]Deleting vbucket: default/52
[couchdb:info,2012-08-24T15:35:58.273,ns_1@10.3.3.116:couch_server:couch_log:info:39]Deleting database default/52
[couchdb:info,2012-08-24T15:35:58.277,ns_1@10.3.3.116:couch_server:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/default/52.couch.1
[couchdb:info,2012-08-24T15:35:58.284,ns_1@10.3.3.116:<0.31051.1>:couch_log:info:39]Database `default/master`, design document `_design/dev_ddoc21` updated (new revision: 2-439e14c7, deleted: false)
[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated
  new signature: 3b5be39b8d4f0063c8435a87369219f5
  shutdown flag: false
  waiting clients: 0

[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal
[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown
[views:debug,2012-08-24T15:35:58.286,ns_1@10.3.3.116:<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update
[ns_server:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false)
[views:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300]
Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>,
                                     {set_view_params,1024,[],[],true}])
[couchdb:error,2012-08-24T15:35:58.294,ns_1@10.3.3.116:<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>,
                                       {not_found,no_db_file},
                                       <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>}
initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]
Comment by Aleksey Kondratenko [ 24/Aug/12 ]
Looks like view engine is trying to open vbucket we just passed to cleanup
Comment by Aleksey Kondratenko [ 24/Aug/12 ]
actually signature is changing. So indeed can be stale index file
Comment by Aleksey Kondratenko [ 24/Aug/12 ]
Hm. So apparently view group is shutdown at:


[couchdb:info,2012-08-24T15:35:55.689,ns_1@10.3.3.116:<0.30300.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: normal
[couchdb:info,2012-08-24T15:35:55.690,ns_1@10.3.3.116:<0.30309.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: shutdown

then after old signature is reused index file is still there


[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated
  new signature: 3b5be39b8d4f0063c8435a87369219f5
  shutdown flag: false
  waiting clients: 0

[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal
[couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown
[views:debug,2012-08-24T15:35:58.286,ns_1@10.3.3.116:<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update
[ns_server:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false)
[views:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300]
Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>,
                                     {set_view_params,1024,[],[],true}])
[couchdb:error,2012-08-24T15:35:58.294,ns_1@10.3.3.116:<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>,
                                       {not_found,no_db_file},
                                       <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>}
initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]

[couchdb:error,2012-08-24T15:35:58.295,ns_1@10.3.3.116:<0.30372.2>:couch_log:error:42]couch_set_view_group error opening set view group `_design/dev_ddoc21`, signature `3b5be39b8d4f0063c8435a87369219f5', from set `default`: {error,
                                                                                                                                           {db_open_error,
                                                                                                                                            <<"default/52">>,
                                                                                                                                            {not_found,
                                                                                                                                             no_db_file},
                                                                                                                                            <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>}}

and just before that default/52 is deleted.

Maybe you already aware of that. We either need to run cleanup before each ddoc update or your code needs to guard itself against conditions like that.
Comment by Thuan Nguyen [ 25/Aug/12 ]
Integrated in github-couchdb-preview #491 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/491/])
    MB-6415 Delete index files on view group terminate (Revision ddfada33d94705f0bace58b90cb2b31f75e17c48)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/src/couch_set_view_group.erl
* src/couch_set_view/src/couch_set_view.erl
Comment by Karan Kumar (Inactive) [ 27/Aug/12 ]
Still seeing this failure

ns_server:debug,2012-08-26T18:40:58.862,ns_1@10.3.3.123:<0.21229.8>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.21228.8>} exited with reason {bad_return_value,
                                                                                 {error,
                                                                                  {db_open_error,
                                                                                   <<"default/0">>,
                                                                                   {not_found,
                                                                                    no_db_file},
                                                                                   <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}}
[

[user:info,2012-08-26T18:40:58.880,ns_1@10.3.3.123:<0.8243.9>:ns_orchestrator:handle_info:294]Rebalance exited with reason {noproc,
                              {gen_server,call,
                               [{'ns_memcached-default','ns_1@10.3.3.123'},
                                {get_vbucket,5},
                                60000]}}



=========================CRASH REPORT=========================
  crasher:
    initial call: capi_set_view_manager:init/1
    pid: <0.21228.8>
    registered_name: []
    exception exit: {bad_return_value,
                        {error,
                            {db_open_error,<<"default/0">>,
                                {not_found,no_db_file},
                                <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}}
      in function gen_server:init_it/6
    ancestors: ['ns_memcached_sup-default','single_bucket_sup-default',
                  <0.21206.8>]
Comment by Karan Kumar (Inactive) [ 27/Aug/12 ]
This is on build 1640
Comment by Karan Kumar (Inactive) [ 27/Aug/12 ]
<project name="couchdb" path="couchdb" revision="ddfada33d94705f0bace58b90cb2b31f75e17c48"/>
Comment by Thuan Nguyen [ 27/Aug/12 ]
Integrated in github-couchdb-preview #492 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/492/])
    MB-6415 Delete index files on terminate after ddoc updated (Revision badcdf3bec99c02415bebaca736687b48a7e0784)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/src/couch_set_view_group.erl
* src/couch_set_view/test/02-old-index-cleanup.t
Comment by Filipe Manana [ 27/Aug/12 ]
Karan, try build 1643, this was incomplete before.
Comment by Filipe Manana [ 28/Aug/12 ]
Fixed in master.
Comment by Deepkaran Salooja [ 11/Sep/12 ]
Verified. This failure is no longer there on the latest build 1706.
Generated at Wed Apr 16 19:45:54 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.