Failure to open view group when rebalance is ongoing

Farshid Ghods added a comment - 24/Aug/12 9:39 PM also fails in make simple-test

Aleksey Kondratenko added a comment - 24/Aug/12 10:04 PM Here's what I'm seeing: [views:info,2012-08-24T15:35:58.218, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:353] couch_set_view:set_partition_states([<<"default">>,<<"_design/dev_ddoc21">>,                                      [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,                                       16,17,18,19,20,21,22,23,24,25,26,27,28,                                       29,30,31,32,33,34,35,36,37,38,39,40,41,                                       42,43,44,45,46,47,48,49,50,51,205,206,                                       207,208,209,210,211,212,213,214,215,216,                                       217,218,219,220,221,222,223,224,225,226,                                       227,228,229,230,231,232,233,234,235,236,                                       237,238,239,240,241,242,243,244,245,246,                                       247,248,249,250,251,252,253,254,255,256,                                       257,258,259,260,261,262,263,264,265,266,                                       267,268,269,270,271,272,273,274,275,276,                                       277,278,279,280,281,282,283,284,285,286,                                       287,288,289,290,291,292,293,294,295,296,                                       297,298,299,300,301,302,303,304,305,306,                                       307,308,309,310,311,312,313,314,315,316,                                       317,318,319,320,321,322,323,324,325,326,                                       327,328,329,330,331,332,333,334,335,336,                                       337,338,339,340,341,342,343,344,345,346,                                       347,348,349,350,351,352,353,354,355,356,                                       357,358,359,360,361,362,363,364,365,366,                                       367,368,369,370,371,372,373,374,375,376,                                       377,378,379,380,381,382,383,384,385,386,                                       387,388,389,390,391,392,393,394,395,396,                                       397,398,399,400,401,402,403,404,405,406,                                       407,408,409,410,411,412,413,414,415,416,                                       417,418,419,420,421,422,423,424,425,426,                                       427,428,429,430,431,432,433,434,435,436,                                       437,438,439,440,441,442,443,461,462,463,                                       464,465,466,467,468,469,470,471,472,473,                                       474,475,476,477,478,479,480,481,482,483,                                       484,485,486,487,488,489,490,491,492,493,                                       494,495,496,497,498,499,500,501,502,503,                                       504,505,506,507,508,509,510,511],                                      [],                                      [52,53,54,55,56,57,58,59,60,61,62,63,64,                                       65,66,67,68,69,70,71,72,73,74,75,76,77,                                       78,79,80,81,82,83,84,85,86,87,88,89,90,                                       91,92,93,94,95,96,97,98,99,100,101,102,                                       103,104,105,106,107,108,109,110,111,112,                                       113,114,115,116,117,118,119,120,121,122,                                       123,124,125,126,127,128,129,130,131,132,                                       133,134,135,136,137,138,139,140,141,142,                                       143,144,145,146,147,148,149,150,151,152,                                       153,154,155,156,157,158,159,160,161,162,                                       163,164,165,166,167,168,169,170,171,172,                                       173,174,175,176,177,178,179,180,181,182,                                       183,184,185,186,187,188,189,190,191,192,                                       193,194,195,196,197,198,199,200,201,202,                                       203,204,444,445,446,447,448,449,450,451,                                       452,453,454,455,456,457,458,459,460,512,                                       513,514,515,516,517,518,519,520,521,522,                                       523,524,525,526,527,528,529,530,531,532,                                       533,534,535,536,537,538,539,540,541,542,                                       543,544,545,546,547,548,549,550,551,552,                                       553,554,555,556,557,558,559,560,561,562,                                       563,564,565,566,567,568,569,570,571,572,                                       573,574,575,576,577,578,579,580,581,582,                                       583,584,585,586,587,588,589,590,591,592,                                       593,594,595,596,597,598,599,600,601,602,                                       603,604,605,606,607,608,609,610,611,612,                                       613,614,615,616,617,618,619,620,621,622,                                       623,624,625,626,627,628,629,630,631,632,                                       633,634,635,636,637,638,639,640,641,642,                                       643,644,645,646,647,648,649,650,651,652,                                       653,654,655,656,657,658,659,660,661,662,                                       663,664,665,666,667,668,669,670,671,672,                                       673,674,675,676,677,678,679,680,681,682,                                       683,684,685,686,687,688,689,690,691,692,                                       693,694,695,696,697,698,699,700,701,702,                                       703,704,705,706,707,708,709,710,711,712,                                       713,714,715,716,717,718,719,720,721,722,                                       723,724,725,726,727,728,729,730,731,732,                                       733,734,735,736,737,738,739,740,741,742,                                       743,744,745,746,747,748,749,750,751,752,                                       753,754,755,756,757,758,759,760,761,762,                                       763,764,765,766,767,768,769,770,771,772,                                       773,774,775,776,777,778,779,780,781,782,                                       783,784,785,786,787,788,789,790,791,792,                                       793,794,795,796,797,798,799,800,801,802,                                       803,804,805,806,807,808,809,810,811,812,                                       813,814,815,816,817,818,819,820,821,822,                                       823,824,825,826,827,828,829,830,831,832,                                       833,834,835,836,837,838,839,840,841,842,                                       843,844,845,846,847,848,849,850,851,852,                                       853,854,855,856,857,858,859,860,861,862,                                       863,864,865,866,867,868,869,870,871,872,                                       873,874,875,876,877,878,879,880,881,882,                                       883,884,885,886,887,888,889,890,891,892,                                       893,894,895,896,897,898,899,900,901,902,                                       903,904,905,906,907,908,909,910,911,912,                                       913,914,915,916,917,918,919,920,921,922,                                       923,924,925,926,927,928,929,930,931,932,                                       933,934,935,936,937,938,939,940,941,942,                                       943,944,945,946,947,948,949,950,951,952,                                       953,954,955,956,957,958,959,960,961,962,                                       963,964,965,966,967,968,969,970,971,972,                                       973,974,975,976,977,978,979,980,981,982,                                       983,984,985,986,987,988,989,990,991,992,                                       993,994,995,996,997,998,999,1000,1001,                                       1002,1003,1004,1005,1006,1007,1008,1009,                                       1010,1011,1012,1013,1014,1015,1016,1017,                                       1018,1019,1020,1021,1022,1023]]) returned ok in 30ms [views:info,2012-08-24T15:35:58.221, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:357] Calling couch_set_view:add_replica_partitions([<<"default">>,                                                <<"_design/dev_ddoc21">>,                                                [53,54,55,56,57,58,59,60,61,62,                                                 63,64,65,66,67,68,69,70,71,72,                                                 73,74,75,76,77,78,79,80,81,82,                                                 83,84,85,86,87,88,89,90,91,92,                                                 93,94,95,96,97,98,99,100,101,                                                 102,103,104,105,106,107,108,                                                 109,110,111,112,113,114,115,                                                 116,117,118,119,120,121,122,                                                 123,124,125,126,127,128,129,                                                 130,131,132,133,134,135,136,                                                 137,138,139,140,141,142,143,                                                 144,145,146,147,148,149,150,                                                 151,152,153,154,155,156,157,                                                 158,159,160,161,162,163,164,                                                 165,166,167,168,169,170,171,                                                 172,173,174,175,176,177,178,                                                 179,180,181,182,183,184,185,                                                 186,187,188,189,190,191,192,                                                 193,194,195,196,197,198,199,                                                 200,201,202,203,204,444,445,                                                 446,447,448,449,450,451,452,                                                 453,454,455,456,457,458,459,                                                 460,512,546,547,548,549,550,                                                 551,552,553,554,555,556,557,                                                 558,559,560,561,562,563,564,                                                 598,599,600,601,602,603,604,                                                 605,606,607,608,609,610,611,                                                 612,613,614,666,667,668,669,                                                 670,671,672,673,674,675,676,                                                 677,678,679,680,681,682,683,                                                 684,685,686,687,688,689,690,                                                 691,692,693,694,695,696,697,                                                 698,699,700,701,702,703,704,                                                 705,706,707,708,709,710,711,                                                 712,713,714,715,716,717,718,                                                 719,720,721,722,723,724,725,                                                 726,727,728,729,730,731,732,                                                 768,871,872,873,874,875,876,                                                 877,878,879,880,881,882,883,                                                 884,885,886,887,888,889,890,                                                 891,892,893,894,895,896,897,                                                 898,899,900,901,902,903,904,                                                 905,906,907,908,909,910,911,                                                 912,913,914,915,916,917,918,                                                 919,920,921,922,923,924,925,                                                 926,927,928,929,930,931,932,                                                 933,934,935,936,937]]) [couchdb:info,2012-08-24T15:35:58.222, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, defined new replica partitions: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] New full set of replica partitions is: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] [views:info,2012-08-24T15:35:58.246, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:358] couch_set_view:add_replica_partitions([<<"default">>,<<"_design/dev_ddoc21">>,                                        [53,54,55,56,57,58,59,60,61,62,63,64,                                         65,66,67,68,69,70,71,72,73,74,75,76,                                         77,78,79,80,81,82,83,84,85,86,87,88,                                         89,90,91,92,93,94,95,96,97,98,99,100,                                         101,102,103,104,105,106,107,108,109,                                         110,111,112,113,114,115,116,117,118,                                         119,120,121,122,123,124,125,126,127,                                         128,129,130,131,132,133,134,135,136,                                         137,138,139,140,141,142,143,144,145,                                         146,147,148,149,150,151,152,153,154,                                         155,156,157,158,159,160,161,162,163,                                         164,165,166,167,168,169,170,171,172,                                         173,174,175,176,177,178,179,180,181,                                         182,183,184,185,186,187,188,189,190,                                         191,192,193,194,195,196,197,198,199,                                         200,201,202,203,204,444,445,446,447,                                         448,449,450,451,452,453,454,455,456,                                         457,458,459,460,512,546,547,548,549,                                         550,551,552,553,554,555,556,557,558,                                         559,560,561,562,563,564,598,599,600,                                         601,602,603,604,605,606,607,608,609,                                         610,611,612,613,614,666,667,668,669,                                         670,671,672,673,674,675,676,677,678,                                         679,680,681,682,683,684,685,686,687,                                         688,689,690,691,692,693,694,695,696,                                         697,698,699,700,701,702,703,704,705,                                         706,707,708,709,710,711,712,713,714,                                         715,716,717,718,719,720,721,722,723,                                         724,725,726,727,728,729,730,731,732,                                         768,871,872,873,874,875,876,877,878,                                         879,880,881,882,883,884,885,886,887,                                         888,889,890,891,892,893,894,895,896,                                         897,898,899,900,901,902,903,904,905,                                         906,907,908,909,910,911,912,913,914,                                         915,916,917,918,919,920,921,922,923,                                         924,925,926,927,928,929,930,931,932,                                         933,934,935,936,937]]) returned ok in 23ms [views:info,2012-08-24T15:35:58.248, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:359] Calling couch_set_view:remove_replica_partitions([<<"default">>,                                                   <<"_design/dev_ddoc21">>,                                                   [52,513,514,515,516,517,518,                                                    519,520,521,522,523,524,                                                    525,526,527,528,529,530,                                                    531,532,533,534,535,536,                                                    537,538,539,540,541,542,                                                    543,544,545,565,566,567,                                                    568,569,570,571,572,573,                                                    574,575,576,577,578,579,                                                    580,581,582,583,584,585,                                                    586,587,588,589,590,591,                                                    592,593,594,595,596,597,                                                    615,616,617,618,619,620,                                                    621,622,623,624,625,626,                                                    627,628,629,630,631,632,                                                    633,634,635,636,637,638,                                                    639,640,641,642,643,644,                                                    645,646,647,648,649,650,                                                    651,652,653,654,655,656,                                                    657,658,659,660,661,662,                                                    663,664,665,733,734,735,                                                    736,737,738,739,740,741,                                                    742,743,744,745,746,747,                                                    748,749,750,751,752,753,                                                    754,755,756,757,758,759,                                                    760,761,762,763,764,765,                                                    766,767,769,770,771,772,                                                    773,774,775,776,777,778,                                                    779,780,781,782,783,784,                                                    785,786,787,788,789,790,                                                    791,792,793,794,795,796,                                                    797,798,799,800,801,802,                                                    803,804,805,806,807,808,                                                    809,810,811,812,813,814,                                                    815,816,817,818,819,820,                                                    821,822,823,824,825,826,                                                    827,828,829,830,831,832,                                                    833,834,835,836,837,838,                                                    839,840,841,842,843,844,                                                    845,846,847,848,849,850,                                                    851,852,853,854,855,856,                                                    857,858,859,860,861,862,                                                    863,864,865,866,867,868,                                                    869,870,938,939,940,941,                                                    942,943,944,945,946,947,                                                    948,949,950,951,952,953,                                                    954,955,956,957,958,959,                                                    960,961,962,963,964,965,                                                    966,967,968,969,970,971,                                                    972,973,974,975,976,977,                                                    978,979,980,981,982,983,                                                    984,985,986,987,988,989,                                                    990,991,992,993,994,995,                                                    996,997,998,999,1000,1001,                                                    1002,1003,1004,1005,1006,                                                    1007,1008,1009,1010,1011,                                                    1012,1013,1014,1015,1016,                                                    1017,1018,1019,1020,1021,                                                    1022,1023]]) [couchdb:info,2012-08-24T15:35:58.269, ns_1@10.3.3.116 :<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, partition states updated active partitions before: [] active partitions after: [] passive partitions before: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] cleanup partitions before: [] cleanup partitions after: [52] unindexable partitions: [] replica partitions before: [] replica partitions after: [] replicas on transfer before: [] replicas on transfer after: [] pending transition before:   active: []   passive: [] pending transition after:   active: []   passive: [] [couchdb:info,2012-08-24T15:35:58.270, ns_1@10.3.3.116 :<0.30346.2>:couch_log:info:39]Started cleanup process <0.30364.2> for set view `default`, replica group `_design/dev_ddoc21` [couchdb:info,2012-08-24T15:35:58.271, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, marked the following replica partitions for removal: [52,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023] [couchdb:info,2012-08-24T15:35:58.271, ns_1@10.3.3.116 :<0.30346.2>:couch_log:info:39]Cleanup finished for set view `default`, replica group `_design/dev_ddoc21` Removed 0 values from the index in 0.000 seconds active partitions before: [] active partitions after: [] passive partitions before: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] cleanup partitions before: [52] cleanup partitions after: [] [views:info,2012-08-24T15:35:58.271, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:360] couch_set_view:remove_replica_partitions([<<"default">>,                                           <<"_design/dev_ddoc21">>,                                           [52,513,514,515,516,517,518,519,520,                                            521,522,523,524,525,526,527,528,                                            529,530,531,532,533,534,535,536,                                            537,538,539,540,541,542,543,544,                                            545,565,566,567,568,569,570,571,                                            572,573,574,575,576,577,578,579,                                            580,581,582,583,584,585,586,587,                                            588,589,590,591,592,593,594,595,                                            596,597,615,616,617,618,619,620,                                            621,622,623,624,625,626,627,628,                                            629,630,631,632,633,634,635,636,                                            637,638,639,640,641,642,643,644,                                            645,646,647,648,649,650,651,652,                                            653,654,655,656,657,658,659,660,                                            661,662,663,664,665,733,734,735,                                            736,737,738,739,740,741,742,743,                                            744,745,746,747,748,749,750,751,                                            752,753,754,755,756,757,758,759,                                            760,761,762,763,764,765,766,767,                                            769,770,771,772,773,774,775,776,                                            777,778,779,780,781,782,783,784,                                            785,786,787,788,789,790,791,792,                                            793,794,795,796,797,798,799,800,                                            801,802,803,804,805,806,807,808,                                            809,810,811,812,813,814,815,816,                                            817,818,819,820,821,822,823,824,                                            825,826,827,828,829,830,831,832,                                            833,834,835,836,837,838,839,840,                                            841,842,843,844,845,846,847,848,                                            849,850,851,852,853,854,855,856,                                            857,858,859,860,861,862,863,864,                                            865,866,867,868,869,870,938,939,                                            940,941,942,943,944,945,946,947,                                            948,949,950,951,952,953,954,955,                                            956,957,958,959,960,961,962,963,                                            964,965,966,967,968,969,970,971,                                            972,973,974,975,976,977,978,979,                                            980,981,982,983,984,985,986,987,                                            988,989,990,991,992,993,994,995,                                            996,997,998,999,1000,1001,1002,                                            1003,1004,1005,1006,1007,1008,1009,                                            1010,1011,1012,1013,1014,1015,1016,                                            1017,1018,1019,1020,1021,1022,                                            1023]]) returned ok in 21ms [ns_server:debug,2012-08-24T15:35:58.273, ns_1@10.3.3.116 :'capi_ddoc_replication_srv-default':cb_generic_replication_srv:handle_cast:131]Writing replicated ddoc {doc,<<"_design/dev_ddoc21">>,                              {2,<<67,158,20,199>>},                              <<"{\"views\":{\"views0\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"},\"views1\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"}}}">>,                              0,false,[]} [ns_server:info,2012-08-24T15:35:58.273, ns_1@10.3.3.116 :<0.31061.1>:mc_connection:do_delete_vbucket:123]Deleting vbucket: default/52 [couchdb:info,2012-08-24T15:35:58.273, ns_1@10.3.3.116 :couch_server:couch_log:info:39]Deleting database default/52 [couchdb:info,2012-08-24T15:35:58.277, ns_1@10.3.3.116 :couch_server:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/default/52.couch.1 [couchdb:info,2012-08-24T15:35:58.284, ns_1@10.3.3.116 :<0.31051.1>:couch_log:info:39]Database `default/master`, design document `_design/dev_ddoc21` updated (new revision: 2-439e14c7, deleted: false) [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated   new signature: 3b5be39b8d4f0063c8435a87369219f5   shutdown flag: false   waiting clients: 0 [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown [views:debug,2012-08-24T15:35:58.286, ns_1@10.3.3.116 :<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update [ns_server:info,2012-08-24T15:35:58.287, ns_1@10.3.3.116 :'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false) [views:info,2012-08-24T15:35:58.287, ns_1@10.3.3.116 :'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300] Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>,                                      {set_view_params,1024,[],[],true}]) [couchdb:error,2012-08-24T15:35:58.294, ns_1@10.3.3.116 :<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>,                                        {not_found,no_db_file},                                        <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>} initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937]

Aleksey Kondratenko added a comment - 24/Aug/12 10:05 PM Looks like view engine is trying to open vbucket we just passed to cleanup

Aleksey Kondratenko added a comment - 24/Aug/12 11:06 PM actually signature is changing. So indeed can be stale index file

Aleksey Kondratenko added a comment - 24/Aug/12 11:09 PM Hm. So apparently view group is shutdown at: [couchdb:info,2012-08-24T15:35:55.689, ns_1@10.3.3.116 :<0.30300.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:55.690, ns_1@10.3.3.116 :<0.30309.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: shutdown then after old signature is reused index file is still there [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated   new signature: 3b5be39b8d4f0063c8435a87369219f5   shutdown flag: false   waiting clients: 0 [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:58.285, ns_1@10.3.3.116 :<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown [views:debug,2012-08-24T15:35:58.286, ns_1@10.3.3.116 :<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update [ns_server:info,2012-08-24T15:35:58.287, ns_1@10.3.3.116 :'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false) [views:info,2012-08-24T15:35:58.287, ns_1@10.3.3.116 :'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300] Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>,                                      {set_view_params,1024,[],[],true}]) [couchdb:error,2012-08-24T15:35:58.294, ns_1@10.3.3.116 :<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>,                                        {not_found,no_db_file},                                        <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>} initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] [couchdb:error,2012-08-24T15:35:58.295, ns_1@10.3.3.116 :<0.30372.2>:couch_log:error:42]couch_set_view_group error opening set view group `_design/dev_ddoc21`, signature `3b5be39b8d4f0063c8435a87369219f5', from set `default`: {error,                                                                                                                                            {db_open_error,                                                                                                                                             <<"default/52">>,                                                                                                                                             {not_found,                                                                                                                                              no_db_file},                                                                                                                                             <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>}} and just before that default/52 is deleted. Maybe you already aware of that. We either need to run cleanup before each ddoc update or your code needs to guard itself against conditions like that.

Thuan Nguyen added a comment - 25/Aug/12 4:41 PM Integrated in github-couchdb-preview #491 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/491/ ])      MB-6415 Delete index files on view group terminate (Revision ddfada33d94705f0bace58b90cb2b31f75e17c48)      Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl * src/couch_set_view/src/couch_set_view.erl

Karan Kumar added a comment - 27/Aug/12 1:42 AM Still seeing this failure ns_server:debug,2012-08-26T18:40:58.862, ns_1@10.3.3.123 :<0.21229.8>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.21228.8>} exited with reason {bad_return_value,                                                                                  {error,                                                                                   {db_open_error,                                                                                    <<"default/0">>,                                                                                    {not_found,                                                                                     no_db_file},                                                                                    <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}} [ [user:info,2012-08-26T18:40:58.880, ns_1@10.3.3.123 :<0.8243.9>:ns_orchestrator:handle_info:294]Rebalance exited with reason {noproc,                               {gen_server,call,                                [{'ns_memcached-default',' ns_1@10.3.3.123 '},                                 {get_vbucket,5},                                 60000]}} =========================CRASH REPORT=========================   crasher:     initial call: capi_set_view_manager:init/1     pid: <0.21228.8>     registered_name: []     exception exit: {bad_return_value,                         {error,                             {db_open_error,<<"default/0">>,                                 {not_found,no_db_file},                                 <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}}       in function gen_server:init_it/6     ancestors: ['ns_memcached_sup-default','single_bucket_sup-default',                   <0.21206.8>]

Karan Kumar added a comment - 27/Aug/12 1:49 AM This is on build 1640

Karan Kumar added a comment - 27/Aug/12 1:50 AM <project name="couchdb" path="couchdb" revision="ddfada33d94705f0bace58b90cb2b31f75e17c48"/>

Thuan Nguyen added a comment - 27/Aug/12 6:45 AM Integrated in github-couchdb-preview #492 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/492/ ])      MB-6415 Delete index files on terminate after ddoc updated (Revision badcdf3bec99c02415bebaca736687b48a7e0784)      Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl * src/couch_set_view/test/02-old-index-cleanup.t

Filipe Manana added a comment - 27/Aug/12 8:00 AM Karan, try build 1643, this was incomplete before.

Filipe Manana added a comment - 28/Aug/12 4:32 AM Fixed in master.

Deepkaran Salooja added a comment - 11/Sep/12 5:27 AM Verified. This failure is no longer there on the latest build 1706.