[MB-6415] Failure to open view group when rebalance is ongoing Created: 24/Aug/12 Updated: 11/Sep/12 Resolved: 28/Aug/12 |
|
| Status: | Closed |
| Project: | Couchbase Server |
| Component/s: | view-engine |
| Affects Version/s: | 2.0-beta |
| Fix Version/s: | 2.0 |
| Security Level: | Public |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Karan Kumar | Assignee: | Filipe Manana |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | regression | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: | 2.0.0-1633-rel | ||
| Attachments: |
|
| Description |
|
There are many other bugs around the same area, but they seem to be timeout related.
Not completely sure, if this is same as them. Filing for being safe. http://www.couchbase.com/issues/browse/MB-6284 http://www.couchbase.com/issues/browse/MB-5978 [rebalance:debug,2012-08-24T15:35:58.601,ns_1@10.3.3.123:<0.151.24>:janitor_agent:bulk_set_vbucket_state:374]bulk vbucket state change failed for: [{'ns_1@10.3.3.120', {'EXIT', {killed, {gen_server,call, [{'janitor_agent-default','ns_1@10.3.3.120'}, {if_rebalance,<0.32709.23>, {update_vbucket_state,70,replica,undefined,undefined}}, infinity]}}}}] [error_logger:error,2012-08-24T15:35:58.829,ns_1@10.3.3.123:error_logger:ale_error_logger_handler:log_report:72] =========================CRASH REPORT========================= crasher: initial call: ns_vbucket_mover:init/1 pid: <0.32709.23> registered_name: [] exception exit: {{bulk_set_vbucket_state_failed, [{'ns_1@10.3.3.123', {'EXIT', {killed, {gen_server,call, [{'janitor_agent-default','ns_1@10.3.3.123'}, {if_rebalance,<0.32709.23>, {update_vbucket_state,0,replica,undefined, 'ns_1@10.3.3.120'}}, infinity]}}}}]}, [{janitor_agent,bulk_set_vbucket_state,4}, {ns_vbucket_mover,update_replication_post_move,3}, {ns_vbucket_mover,handle_info,2}, {gen_server,handle_msg,5}, {proc_lib,init_p_do_apply,3}]} in function gen_server:terminate/6 ancestors: [<0.32629.23>] messages: [{'EXIT',<0.32716.23>,normal}, {'EXIT',<0.151.24>, {bulk_set_vbucket_state_failed, [{'ns_1@10.3.3.120', {'EXIT', {killed, {gen_server,call, [{'janitor_agent-default','ns_1@10.3.3.120'}, {if_rebalance,<0.32709.23>, {update_vbucket_state,70,replica,undefined, undefined}}, infinity]}}}}]}}, {'EXIT',<0.98.24>, {killed, {gen_server,call, [{'janitor_agent-default','ns_1@10.3.3.120'}, {if_rebalance,<0.32709.23>, {update_vbucket_state,53,active,undefined,undefined}}, infinity]}}}] links: [<0.32629.23>,<0.32714.23>,<0.62.0>] dictionary: [{bucket_name,"default"}, {i_am_master_mover,true}, {child_processes,[<0.151.24>,<0.98.24>,<0.32718.23>, <0.32717.23>,<0.32716.23>]}] |
| Comments |
| Comment by Farshid Ghods [ 24/Aug/12 ] |
| also fails in make simple-test |
| Comment by Aleksey Kondratenko [ 24/Aug/12 ] |
|
Here's what I'm seeing: [views:info,2012-08-24T15:35:58.218,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:353] couch_set_view:set_partition_states([<<"default">>,<<"_design/dev_ddoc21">>, [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15, 16,17,18,19,20,21,22,23,24,25,26,27,28, 29,30,31,32,33,34,35,36,37,38,39,40,41, 42,43,44,45,46,47,48,49,50,51,205,206, 207,208,209,210,211,212,213,214,215,216, 217,218,219,220,221,222,223,224,225,226, 227,228,229,230,231,232,233,234,235,236, 237,238,239,240,241,242,243,244,245,246, 247,248,249,250,251,252,253,254,255,256, 257,258,259,260,261,262,263,264,265,266, 267,268,269,270,271,272,273,274,275,276, 277,278,279,280,281,282,283,284,285,286, 287,288,289,290,291,292,293,294,295,296, 297,298,299,300,301,302,303,304,305,306, 307,308,309,310,311,312,313,314,315,316, 317,318,319,320,321,322,323,324,325,326, 327,328,329,330,331,332,333,334,335,336, 337,338,339,340,341,342,343,344,345,346, 347,348,349,350,351,352,353,354,355,356, 357,358,359,360,361,362,363,364,365,366, 367,368,369,370,371,372,373,374,375,376, 377,378,379,380,381,382,383,384,385,386, 387,388,389,390,391,392,393,394,395,396, 397,398,399,400,401,402,403,404,405,406, 407,408,409,410,411,412,413,414,415,416, 417,418,419,420,421,422,423,424,425,426, 427,428,429,430,431,432,433,434,435,436, 437,438,439,440,441,442,443,461,462,463, 464,465,466,467,468,469,470,471,472,473, 474,475,476,477,478,479,480,481,482,483, 484,485,486,487,488,489,490,491,492,493, 494,495,496,497,498,499,500,501,502,503, 504,505,506,507,508,509,510,511], [], [52,53,54,55,56,57,58,59,60,61,62,63,64, 65,66,67,68,69,70,71,72,73,74,75,76,77, 78,79,80,81,82,83,84,85,86,87,88,89,90, 91,92,93,94,95,96,97,98,99,100,101,102, 103,104,105,106,107,108,109,110,111,112, 113,114,115,116,117,118,119,120,121,122, 123,124,125,126,127,128,129,130,131,132, 133,134,135,136,137,138,139,140,141,142, 143,144,145,146,147,148,149,150,151,152, 153,154,155,156,157,158,159,160,161,162, 163,164,165,166,167,168,169,170,171,172, 173,174,175,176,177,178,179,180,181,182, 183,184,185,186,187,188,189,190,191,192, 193,194,195,196,197,198,199,200,201,202, 203,204,444,445,446,447,448,449,450,451, 452,453,454,455,456,457,458,459,460,512, 513,514,515,516,517,518,519,520,521,522, 523,524,525,526,527,528,529,530,531,532, 533,534,535,536,537,538,539,540,541,542, 543,544,545,546,547,548,549,550,551,552, 553,554,555,556,557,558,559,560,561,562, 563,564,565,566,567,568,569,570,571,572, 573,574,575,576,577,578,579,580,581,582, 583,584,585,586,587,588,589,590,591,592, 593,594,595,596,597,598,599,600,601,602, 603,604,605,606,607,608,609,610,611,612, 613,614,615,616,617,618,619,620,621,622, 623,624,625,626,627,628,629,630,631,632, 633,634,635,636,637,638,639,640,641,642, 643,644,645,646,647,648,649,650,651,652, 653,654,655,656,657,658,659,660,661,662, 663,664,665,666,667,668,669,670,671,672, 673,674,675,676,677,678,679,680,681,682, 683,684,685,686,687,688,689,690,691,692, 693,694,695,696,697,698,699,700,701,702, 703,704,705,706,707,708,709,710,711,712, 713,714,715,716,717,718,719,720,721,722, 723,724,725,726,727,728,729,730,731,732, 733,734,735,736,737,738,739,740,741,742, 743,744,745,746,747,748,749,750,751,752, 753,754,755,756,757,758,759,760,761,762, 763,764,765,766,767,768,769,770,771,772, 773,774,775,776,777,778,779,780,781,782, 783,784,785,786,787,788,789,790,791,792, 793,794,795,796,797,798,799,800,801,802, 803,804,805,806,807,808,809,810,811,812, 813,814,815,816,817,818,819,820,821,822, 823,824,825,826,827,828,829,830,831,832, 833,834,835,836,837,838,839,840,841,842, 843,844,845,846,847,848,849,850,851,852, 853,854,855,856,857,858,859,860,861,862, 863,864,865,866,867,868,869,870,871,872, 873,874,875,876,877,878,879,880,881,882, 883,884,885,886,887,888,889,890,891,892, 893,894,895,896,897,898,899,900,901,902, 903,904,905,906,907,908,909,910,911,912, 913,914,915,916,917,918,919,920,921,922, 923,924,925,926,927,928,929,930,931,932, 933,934,935,936,937,938,939,940,941,942, 943,944,945,946,947,948,949,950,951,952, 953,954,955,956,957,958,959,960,961,962, 963,964,965,966,967,968,969,970,971,972, 973,974,975,976,977,978,979,980,981,982, 983,984,985,986,987,988,989,990,991,992, 993,994,995,996,997,998,999,1000,1001, 1002,1003,1004,1005,1006,1007,1008,1009, 1010,1011,1012,1013,1014,1015,1016,1017, 1018,1019,1020,1021,1022,1023]]) returned ok in 30ms [views:info,2012-08-24T15:35:58.221,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:357] Calling couch_set_view:add_replica_partitions([<<"default">>, <<"_design/dev_ddoc21">>, [53,54,55,56,57,58,59,60,61,62, 63,64,65,66,67,68,69,70,71,72, 73,74,75,76,77,78,79,80,81,82, 83,84,85,86,87,88,89,90,91,92, 93,94,95,96,97,98,99,100,101, 102,103,104,105,106,107,108, 109,110,111,112,113,114,115, 116,117,118,119,120,121,122, 123,124,125,126,127,128,129, 130,131,132,133,134,135,136, 137,138,139,140,141,142,143, 144,145,146,147,148,149,150, 151,152,153,154,155,156,157, 158,159,160,161,162,163,164, 165,166,167,168,169,170,171, 172,173,174,175,176,177,178, 179,180,181,182,183,184,185, 186,187,188,189,190,191,192, 193,194,195,196,197,198,199, 200,201,202,203,204,444,445, 446,447,448,449,450,451,452, 453,454,455,456,457,458,459, 460,512,546,547,548,549,550, 551,552,553,554,555,556,557, 558,559,560,561,562,563,564, 598,599,600,601,602,603,604, 605,606,607,608,609,610,611, 612,613,614,666,667,668,669, 670,671,672,673,674,675,676, 677,678,679,680,681,682,683, 684,685,686,687,688,689,690, 691,692,693,694,695,696,697, 698,699,700,701,702,703,704, 705,706,707,708,709,710,711, 712,713,714,715,716,717,718, 719,720,721,722,723,724,725, 726,727,728,729,730,731,732, 768,871,872,873,874,875,876, 877,878,879,880,881,882,883, 884,885,886,887,888,889,890, 891,892,893,894,895,896,897, 898,899,900,901,902,903,904, 905,906,907,908,909,910,911, 912,913,914,915,916,917,918, 919,920,921,922,923,924,925, 926,927,928,929,930,931,932, 933,934,935,936,937]]) [couchdb:info,2012-08-24T15:35:58.222,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, defined new replica partitions: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] New full set of replica partitions is: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] [views:info,2012-08-24T15:35:58.246,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:358] couch_set_view:add_replica_partitions([<<"default">>,<<"_design/dev_ddoc21">>, [53,54,55,56,57,58,59,60,61,62,63,64, 65,66,67,68,69,70,71,72,73,74,75,76, 77,78,79,80,81,82,83,84,85,86,87,88, 89,90,91,92,93,94,95,96,97,98,99,100, 101,102,103,104,105,106,107,108,109, 110,111,112,113,114,115,116,117,118, 119,120,121,122,123,124,125,126,127, 128,129,130,131,132,133,134,135,136, 137,138,139,140,141,142,143,144,145, 146,147,148,149,150,151,152,153,154, 155,156,157,158,159,160,161,162,163, 164,165,166,167,168,169,170,171,172, 173,174,175,176,177,178,179,180,181, 182,183,184,185,186,187,188,189,190, 191,192,193,194,195,196,197,198,199, 200,201,202,203,204,444,445,446,447, 448,449,450,451,452,453,454,455,456, 457,458,459,460,512,546,547,548,549, 550,551,552,553,554,555,556,557,558, 559,560,561,562,563,564,598,599,600, 601,602,603,604,605,606,607,608,609, 610,611,612,613,614,666,667,668,669, 670,671,672,673,674,675,676,677,678, 679,680,681,682,683,684,685,686,687, 688,689,690,691,692,693,694,695,696, 697,698,699,700,701,702,703,704,705, 706,707,708,709,710,711,712,713,714, 715,716,717,718,719,720,721,722,723, 724,725,726,727,728,729,730,731,732, 768,871,872,873,874,875,876,877,878, 879,880,881,882,883,884,885,886,887, 888,889,890,891,892,893,894,895,896, 897,898,899,900,901,902,903,904,905, 906,907,908,909,910,911,912,913,914, 915,916,917,918,919,920,921,922,923, 924,925,926,927,928,929,930,931,932, 933,934,935,936,937]]) returned ok in 23ms [views:info,2012-08-24T15:35:58.248,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:359] Calling couch_set_view:remove_replica_partitions([<<"default">>, <<"_design/dev_ddoc21">>, [52,513,514,515,516,517,518, 519,520,521,522,523,524, 525,526,527,528,529,530, 531,532,533,534,535,536, 537,538,539,540,541,542, 543,544,545,565,566,567, 568,569,570,571,572,573, 574,575,576,577,578,579, 580,581,582,583,584,585, 586,587,588,589,590,591, 592,593,594,595,596,597, 615,616,617,618,619,620, 621,622,623,624,625,626, 627,628,629,630,631,632, 633,634,635,636,637,638, 639,640,641,642,643,644, 645,646,647,648,649,650, 651,652,653,654,655,656, 657,658,659,660,661,662, 663,664,665,733,734,735, 736,737,738,739,740,741, 742,743,744,745,746,747, 748,749,750,751,752,753, 754,755,756,757,758,759, 760,761,762,763,764,765, 766,767,769,770,771,772, 773,774,775,776,777,778, 779,780,781,782,783,784, 785,786,787,788,789,790, 791,792,793,794,795,796, 797,798,799,800,801,802, 803,804,805,806,807,808, 809,810,811,812,813,814, 815,816,817,818,819,820, 821,822,823,824,825,826, 827,828,829,830,831,832, 833,834,835,836,837,838, 839,840,841,842,843,844, 845,846,847,848,849,850, 851,852,853,854,855,856, 857,858,859,860,861,862, 863,864,865,866,867,868, 869,870,938,939,940,941, 942,943,944,945,946,947, 948,949,950,951,952,953, 954,955,956,957,958,959, 960,961,962,963,964,965, 966,967,968,969,970,971, 972,973,974,975,976,977, 978,979,980,981,982,983, 984,985,986,987,988,989, 990,991,992,993,994,995, 996,997,998,999,1000,1001, 1002,1003,1004,1005,1006, 1007,1008,1009,1010,1011, 1012,1013,1014,1015,1016, 1017,1018,1019,1020,1021, 1022,1023]]) [couchdb:info,2012-08-24T15:35:58.269,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, partition states updated active partitions before: [] active partitions after: [] passive partitions before: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] cleanup partitions before: [] cleanup partitions after: [52] unindexable partitions: [] replica partitions before: [] replica partitions after: [] replicas on transfer before: [] replicas on transfer after: [] pending transition before: active: [] passive: [] pending transition after: active: [] passive: [] [couchdb:info,2012-08-24T15:35:58.270,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Started cleanup process <0.30364.2> for set view `default`, replica group `_design/dev_ddoc21` [couchdb:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, marked the following replica partitions for removal: [52,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023] [couchdb:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Cleanup finished for set view `default`, replica group `_design/dev_ddoc21` Removed 0 values from the index in 0.000 seconds active partitions before: [] active partitions after: [] passive partitions before: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] passive partitions after: [53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] cleanup partitions before: [52] cleanup partitions after: [] [views:info,2012-08-24T15:35:58.271,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:360] couch_set_view:remove_replica_partitions([<<"default">>, <<"_design/dev_ddoc21">>, [52,513,514,515,516,517,518,519,520, 521,522,523,524,525,526,527,528, 529,530,531,532,533,534,535,536, 537,538,539,540,541,542,543,544, 545,565,566,567,568,569,570,571, 572,573,574,575,576,577,578,579, 580,581,582,583,584,585,586,587, 588,589,590,591,592,593,594,595, 596,597,615,616,617,618,619,620, 621,622,623,624,625,626,627,628, 629,630,631,632,633,634,635,636, 637,638,639,640,641,642,643,644, 645,646,647,648,649,650,651,652, 653,654,655,656,657,658,659,660, 661,662,663,664,665,733,734,735, 736,737,738,739,740,741,742,743, 744,745,746,747,748,749,750,751, 752,753,754,755,756,757,758,759, 760,761,762,763,764,765,766,767, 769,770,771,772,773,774,775,776, 777,778,779,780,781,782,783,784, 785,786,787,788,789,790,791,792, 793,794,795,796,797,798,799,800, 801,802,803,804,805,806,807,808, 809,810,811,812,813,814,815,816, 817,818,819,820,821,822,823,824, 825,826,827,828,829,830,831,832, 833,834,835,836,837,838,839,840, 841,842,843,844,845,846,847,848, 849,850,851,852,853,854,855,856, 857,858,859,860,861,862,863,864, 865,866,867,868,869,870,938,939, 940,941,942,943,944,945,946,947, 948,949,950,951,952,953,954,955, 956,957,958,959,960,961,962,963, 964,965,966,967,968,969,970,971, 972,973,974,975,976,977,978,979, 980,981,982,983,984,985,986,987, 988,989,990,991,992,993,994,995, 996,997,998,999,1000,1001,1002, 1003,1004,1005,1006,1007,1008,1009, 1010,1011,1012,1013,1014,1015,1016, 1017,1018,1019,1020,1021,1022, 1023]]) returned ok in 21ms [ns_server:debug,2012-08-24T15:35:58.273,ns_1@10.3.3.116:'capi_ddoc_replication_srv-default':cb_generic_replication_srv:handle_cast:131]Writing replicated ddoc {doc,<<"_design/dev_ddoc21">>, {2,<<67,158,20,199>>}, <<"{\"views\":{\"views0\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"},\"views1\":{\"map\":\"function (doc) { emit(doc.age, doc.first_name);}\"}}}">>, 0,false,[]} [ns_server:info,2012-08-24T15:35:58.273,ns_1@10.3.3.116:<0.31061.1>:mc_connection:do_delete_vbucket:123]Deleting vbucket: default/52 [couchdb:info,2012-08-24T15:35:58.273,ns_1@10.3.3.116:couch_server:couch_log:info:39]Deleting database default/52 [couchdb:info,2012-08-24T15:35:58.277,ns_1@10.3.3.116:couch_server:couch_log:info:39]Deleting file /opt/couchbase/var/lib/couchbase/data/default/52.couch.1 [couchdb:info,2012-08-24T15:35:58.284,ns_1@10.3.3.116:<0.31051.1>:couch_log:info:39]Database `default/master`, design document `_design/dev_ddoc21` updated (new revision: 2-439e14c7, deleted: false) [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated new signature: 3b5be39b8d4f0063c8435a87369219f5 shutdown flag: false waiting clients: 0 [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown [views:debug,2012-08-24T15:35:58.286,ns_1@10.3.3.116:<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update [ns_server:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false) [views:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300] Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>, {set_view_params,1024,[],[],true}]) [couchdb:error,2012-08-24T15:35:58.294,ns_1@10.3.3.116:<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>, {not_found,no_db_file}, <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>} initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] |
| Comment by Aleksey Kondratenko [ 24/Aug/12 ] |
| Looks like view engine is trying to open vbucket we just passed to cleanup |
| Comment by Aleksey Kondratenko [ 24/Aug/12 ] |
| actually signature is changing. So indeed can be stale index file |
| Comment by Aleksey Kondratenko [ 24/Aug/12 ] |
|
Hm. So apparently view group is shutdown at:
[couchdb:info,2012-08-24T15:35:55.689,ns_1@10.3.3.116:<0.30300.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:55.690,ns_1@10.3.3.116:<0.30309.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc20`, signature `3b5be39b8d4f0063c8435a87369219f5`, terminating with reason: shutdown then after old signature is reused index file is still there [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376', design document was updated new signature: 3b5be39b8d4f0063c8435a87369219f5 shutdown flag: false waiting clients: 0 [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30340.2>:couch_log:info:39]Set view `default`, main group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: normal [couchdb:info,2012-08-24T15:35:58.285,ns_1@10.3.3.116:<0.30346.2>:couch_log:info:39]Set view `default`, replica group `_design/dev_ddoc21`, signature `63cea0744ec01dbac6dc69cc22bc0376`, terminating with reason: shutdown [views:debug,2012-08-24T15:35:58.286,ns_1@10.3.3.116:<0.31058.1>:capi_set_view_manager:master_db_watcher:434]Got change in `_design/dev_ddoc21` design document. Initiating set view update [ns_server:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:handle_info:236]Processing update_ddoc _design/dev_ddoc21 (false) [views:info,2012-08-24T15:35:58.287,ns_1@10.3.3.116:'capi_set_view_manager-default':capi_set_view_manager:maybe_define_group:300] Calling couch_set_view:define_group([<<"default">>,<<"_design/dev_ddoc21">>, {set_view_params,1024,[],[],true}]) [couchdb:error,2012-08-24T15:35:58.294,ns_1@10.3.3.116:<0.30377.2>:couch_log:error:42]Error opening database set `default`: {db_open_error,<<"default/52">>, {not_found,no_db_file}, <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>} initial partitions: [52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,512,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,768,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937] [couchdb:error,2012-08-24T15:35:58.295,ns_1@10.3.3.116:<0.30372.2>:couch_log:error:42]couch_set_view_group error opening set view group `_design/dev_ddoc21`, signature `3b5be39b8d4f0063c8435a87369219f5', from set `default`: {error, {db_open_error, <<"default/52">>, {not_found, no_db_file}, <<"Couldn't open database `default/52`, reason: {not_found,no_db_file}">>}} and just before that default/52 is deleted. Maybe you already aware of that. We either need to run cleanup before each ddoc update or your code needs to guard itself against conditions like that. |
| Comment by Thuan Nguyen [ 25/Aug/12 ] |
|
Integrated in github-couchdb-preview #491 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/491/]) Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl * src/couch_set_view/src/couch_set_view.erl |
| Comment by Karan Kumar [ 27/Aug/12 ] |
|
Still seeing this failure
ns_server:debug,2012-08-26T18:40:58.862,ns_1@10.3.3.123:<0.21229.8>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.21228.8>} exited with reason {bad_return_value, {error, {db_open_error, <<"default/0">>, {not_found, no_db_file}, <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}} [ [user:info,2012-08-26T18:40:58.880,ns_1@10.3.3.123:<0.8243.9>:ns_orchestrator:handle_info:294]Rebalance exited with reason {noproc, {gen_server,call, [{'ns_memcached-default','ns_1@10.3.3.123'}, {get_vbucket,5}, 60000]}} =========================CRASH REPORT========================= crasher: initial call: capi_set_view_manager:init/1 pid: <0.21228.8> registered_name: [] exception exit: {bad_return_value, {error, {db_open_error,<<"default/0">>, {not_found,no_db_file}, <<"Couldn't open database `default/0`, reason: {not_found,no_db_file}">>}}} in function gen_server:init_it/6 ancestors: ['ns_memcached_sup-default','single_bucket_sup-default', <0.21206.8>] |
| Comment by Karan Kumar [ 27/Aug/12 ] |
| This is on build 1640 |
| Comment by Karan Kumar [ 27/Aug/12 ] |
| <project name="couchdb" path="couchdb" revision="ddfada33d94705f0bace58b90cb2b31f75e17c48"/> |
| Comment by Thuan Nguyen [ 27/Aug/12 ] |
|
Integrated in github-couchdb-preview #492 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/492/]) Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_group.erl * src/couch_set_view/test/02-old-index-cleanup.t |
| Comment by Filipe Manana [ 27/Aug/12 ] |
| Karan, try build 1643, this was incomplete before. |
| Comment by Filipe Manana [ 28/Aug/12 ] |
| Fixed in master. |
| Comment by Deepkaran Salooja [ 11/Sep/12 ] |
| Verified. This failure is no longer there on the latest build 1706. |