http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6494 Couchbase Server Cluster information: <br/> - 11 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 10 GB RAM and 150 GB disk. <br/> - 8 GB RAM for couchbase server at each node <br/> - Each server has its own drive, no disk sharing with other server. <br/> - Cluster has 2 buckets, default (3GB) and saslbucket (3GB) <br/> - Each bucket has one doc and 2 views for each doc. <br/> - Loader 10.3.2.4 <br/> <br/> * Build 2.0.0-1645 on 11 nodes <br/> * Create cluster with 10 nodes <br/> 10.3.121.13 <br/> 10.3.121.14 <br/> 10.3.121.15 <br/> 10.3.121.16 <br/> 10.3.121.17 <br/> 10.3.121.20 <br/> 10.3.121.22 <br/> 10.3.121.24 <br/> 10.3.121.25 <br/> 10.3.121.23 <br/> * Data path /data <br/> * View path /data <br/> <br/> * Activities testing: <br/> - Do failover, rebalance, swap rebalance in and out. <br/> <br/> * Latest step failed for this bug. <br/> - Failover node 14 <br/> - Rebalance and stop rebalance. So node 14 was kicked out. <br/> - Add node 14 back to cluster and failover node 13. <br/> - Rebalance. Rebalance failed and during rebalance, got error point to node 16 <br/> <br/> &nbsp;&quot;Got error while trying to send close confirmation: {error,enotconn} 13:56:20 - Thu Aug 30, 2012 &quot; on node 16. <br/> <br/> Analyze diags of node 16, see this error at time 2012-08-30T13:56:20 <br/> <br/> [ns_server:debug,2012-08-30T13:56:20.614,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:kill_tapname:932]killing tap named: replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; <br/> [rebalance:info,2012-08-30T13:56:20.620,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:522]Starting tap stream: <br/> [{vbuckets,[]}, <br/> &nbsp;{checkpoints,[]}, <br/> &nbsp;{name,&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;&gt;&gt;}, <br/> &nbsp;{takeover,false}] <br/> {{&quot;10.3.121.25&quot;,11209}, <br/> &nbsp;{&quot;10.3.121.16&quot;,11209}, <br/> &nbsp;[{vbuckets,[30]}, <br/> &nbsp;&nbsp;{takeover,false}, <br/> &nbsp;&nbsp;{suffix,&quot;building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;}, <br/> &nbsp;&nbsp;{username,&quot;saslbucket&quot;}, <br/> &nbsp;&nbsp;{password,&quot;password&quot;}]} <br/> [rebalance:debug,2012-08-30T13:56:20.623,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:555]upstream_sender pid: &lt;0.13244.35&gt; <br/> [rebalance:info,2012-08-30T13:56:20.665,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 18 <br/> [rebalance:info,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 24 <br/> [rebalance:info,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 25 <br/> [rebalance:error,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:confirm_sent_messages:674]Got error while trying to send close confirmation: {error,enotconn} <br/> <br/> [views:info,2012-08-30T13:56:20.852,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&#39;capi_ddoc_replication_srv-saslbucket&#39;:capi_set_view_manager:apply_index_states:351] <br/> Calling couch_set_view:set_partition_states([&lt;&lt;&quot;saslbucket&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;_design/d11&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[12,13,14,15,16,17,19,20,21,22, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;23,24,25,26,36,37,38,39,40,41, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;42,69,70,71,72,73,84,85,86,87, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;92,93,94,103,104,105,106,107, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;108,109,110,111,112,113,114,142, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;143,150,151,152,153,154,155,156, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;178,179,180,181,182,196,197,218, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;219,220,221,222,223,224,225,226, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;227,228,229,230,231,232,233,234, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;235,236,237,242,243,244,245,246, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;247,248,249,250,251,252,253,265, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;266,267,268,288,289,290,291,309, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;310,311,312,313,314,315,321,322, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;323,333,334,335,336,337,338,339, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;340,341,342,343,344,347,348,349, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;350,351,352,353,354,355,362,363, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;364,365,366,367,379,380,381,382, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;383,384,385,395,396,406,407,408, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;409,410,411,446,447,448,449,450, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;451,452,453,503,504,505,506,507, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;508,509,510,511,517,518,519,520, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;521,522,523,524,603,604,613,614, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;615,632,633,634,635,636,637,695, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;696,748,749,750,811,812,813,843, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;844,845,846,847,848,854,876,916, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;917,918,919,920,927,928,929,930, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;931,932,939,940,945,946,947,948, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;949,950,951,965,966,978,979, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1004,1005,1012,1013,1014,1015, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1016,1017,1018,1019,1020,1021, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1022,1023], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[0,1,2,3,4,5,6,7,8,9,10,11,18,27, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;28,29,30,31,32,33,34,35,43,44, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;45,46,47,48,49,50,51,52,53,54, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;55,56,57,58,59,60,61,62,63,64, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;65,66,67,68,74,75,76,77,78,79, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;80,81,82,83,88,89,90,91,95,96, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;97,98,99,100,101,102,115,116, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;117,118,119,120,121,122,123,124, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;125,126,127,128,129,130,131,132, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;133,134,135,136,137,138,139,140, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;141,144,145,146,147,148,149,157, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;158,159,160,161,162,163,164,165, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;166,167,168,169,170,171,172,173, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;174,175,176,177,183,184,185,186, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;187,188,189,190,191,192,193,194, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;195,198,199,200,201,202,203,204, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;205,206,207,208,209,210,211,212, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;213,214,215,216,217,238,239,240, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;241,254,255,256,257,258,259,260, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;261,262,263,264,269,270,271,272, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;273,274,275,276,277,278,279,280, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;281,282,283,284,285,286,287,292, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;293,294,295,296,297,298,299,300, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;301,302,303,304,305,306,307,308, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;316,317,318,319,320,324,325,326, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;327,328,329,330,331,332,345,346, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;356,357,358,359,360,361,368,369, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;370,371,372,373,374,375,376,377, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;378,386,387,388,389,390,391,392, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;393,394,397,398,399,400,401,402, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;403,404,405,412,413,414,415,416, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;417,418,419,420,421,422,423,424, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;425,426,427,428,429,430,431,432, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;433,434,435,436,437,438,439,440, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;441,442,443,444,445,454,455,456, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;457,458,459,460,461,462,463,464, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;465,466,467,468,469,470,471,472, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;473,474,475,476,477,478,479,480, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;481,482,483,484,485,486,487,488, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;489,490,491,492,493,494,495,496, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;497,498,499,500,501,502,512,513, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;514,515,516,525,526,527,528,529, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;530,531,532,533,534,535,536,537, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;538,539,540,541,542,543,544,545, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;546,547,548,549,550,551,552,553, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;554,555,556,557,558,559,560,561, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;562,563,564,565,566,567,568,569, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;570,571,572,573,574,575,576,577, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;578,579,580,581,582,583,584,585, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;586,587,588,589,590,591,592,593, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;594,595,596,597,598,599,600,601, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;602,605,606,607,608,609,610,611, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;612,616,617,618,619,620,621,622, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;623,624,625,626,627,628,629,630, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;631,638,639,640,641,642,643,644, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;645,646,647,648,649,650,651,652, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;653,654,655,656,657,658,659,660, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;661,662,663,664,665,666,667,668, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;669,670,671,672,673,674,675,676, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;677,678,679,680,681,682,683,684, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;685,686,687,688,689,690,691,692, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;693,694,697,698,699,700,701,702, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;703,704,705,706,707,708,709,710, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;711,712,713,714,715,716,717,718, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;719,720,721,722,723,724,725,726, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;727,728,729,730,731,732,733,734, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;735,736,737,738,739,740,741,742, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;743,744,745,746,747,751,752,753, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;754,755,756,757,758,759,760,761, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;762,763,764,765,766,767,768,769, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;770,771,772,773,774,775,776,777, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;778,779,780,781,782,783,784,785, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;786,787,788,789,790,791,792,793, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;794,795,796,797,798,799,800,801, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;802,803,804,805,806,807,808,809, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;810,814,815,816,817,818,819,820, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;821,822,823,824,825,826,827,828, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;829,830,831,832,833,834,835,836, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;837,838,839,840,841,842,849,850, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;851,852,853,855,856,857,858,859, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;860,861,862,863,864,865,866,867, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;868,869,870,871,872,873,874,875, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;877,878,879,880,881,882,883,884, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;885,886,887,888,889,890,891,892, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;893,894,895,896,897,898,899,900, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;901,902,903,904,905,906,907,908, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;909,910,911,912,913,914,915,921, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;922,923,924,925,926,933,934,935, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;936,937,938,941,942,943,944,952, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;953,954,955,956,957,958,959,960, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;961,962,963,964,967,968,969,970, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;971,972,973,974,975,976,977,980, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;981,982,983,984,985,986,987,988, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;989,990,991,992,993,994,995,996, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;997,998,999,1000,1001,1002,1003, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1006,1007,1008,1009,1010,1011]]) <br/> [ns_server:info,2012-08-30T13:56:20.867,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.674.0&gt;: Thu Aug 30 20:56:20.666298 3: TAP (Consumer) eq_tapq:anon_56 - Failed to reset a vbucket 18. Force disconnect <br/> memcached&lt;0.674.0&gt;: Thu Aug 30 20:56:20.666374 3: TAP (Consumer) eq_tapq:anon_56 - disconnected <br/> <br/> Link to diags of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/11ndoes-1645-reb-failed-close-confirmation-error-enotconn-20120830.tgz">https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/11ndoes-1645-reb-failed-close-confirmation-error-enotconn-20120830.tgz</a> <br/> <br/> Link to atop file of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/atop-files/2.0.0/atop-11ndoes-1645-reb-failed-close-confirmation-error-enotconn-20120830.tgz">https://s3.amazonaws.com/packages.couchbase/atop-files/2.0.0/atop-11ndoes-1645-reb-failed-close-confirmation-error-enotconn-20120830.tgz</a> <br/> centos 6.2 64 bit MB-6494

[longevity] rebalance fails with error "Got error while trying to send close confirmation: {error,enotconn} during rebalance" when adding a failed over node back to the cluster and failing over another node

Bug Major Closed Cannot Reproduce Public Chiyoung Seo Thuan Nguyen Thu, 30 Aug 2012 16:33:25 -0500 Thu, 10 Jan 2013 01:21:34 -0600 Tue, 20 Nov 2012 13:29:45 -0600 2.0-beta 2.0.1 ns_server 0 2 Retrying rebalance results in the same error. <br/> Something really weird happens:<br/> <br/> We have replica building downstream (in this case on node .16) closing connection:<br/> <br/> [error_logger:error,2012-08-30T13:56:20.656,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:error_logger:ale_error_logger_handler:log_msg:76]** Generic server &lt;15202.13236.35&gt; terminating <br/> ** Last message in was {tcp,#Port&lt;15202.1659527&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;128,68,0,0,8,0,0,0,0,0,0,12,0,0,0,2,0,0,0,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,4,0,1,255,0,0,0,0,0,0,0,128,68,0,0,8,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,0,12,0,0,0,4,0,0,0,0,0,0,0,0,0,4,0,1,255,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,0,0,0,2,128,68,0,0,8,0,0,18,0,0,0,12,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,6,0,0,0,0,0,0,0,0,0,4,0,1,255,0,0,0,0,0,0,1,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;128,68,0,0,8,0,0,24,0,0,0,12,0,0,0,8,0,0,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,0,4,0,1,255,0,0,0,0,0,0,1,128,68,0,0,8,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,25,0,0,0,12,0,0,0,10,0,0,0,0,0,0,0,0,0,4,0,1,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;255,0,0,0,0,0,0,1,128,68,0,0,8,0,0,26,0,0,0,12,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,12,0,0,0,0,0,0,0,0,0,4,0,1,255,0,0,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,1,128,68,0,0,8,0,0,31,0,0,0,12,0,0,0,14,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,0,0,0,4,0,1,255,0,0,0,0,0,0,1,128,68,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8,0,0,32,0,0,0,12,0,0,0,16,0,0,0,0,0,0,0,0,0,4,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,1,255,0,0,0,0,0,0,1,128,68,0,0,8,0,0,47,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12,0,0,0,18,0,0,0,0,0,0,0,0,0,4,0,1,255,0,0,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,1,128,68,0,0,8,0,0,48,0,0,0,12,0,0,0,20,0,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,0,0,0,0,0,0,4,0,1,255,0,0,0,0,0,0,1&gt;&gt;}<br/> ** When Server state == {state,#Port&lt;15202.1659527&gt;,#Port&lt;15202.1659523&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#Port&lt;15202.1659528&gt;,#Port&lt;15202.1659524&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;15202.13244.35&gt;,&lt;&lt;&gt;&gt;,&lt;&lt;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{set,0,16,16,8,80,48,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[],[],<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[]},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[],[],[],[],[],[],[],[],[],[],<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[]}}},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-1,false,false,0,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{1346,360180,624142},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;not_started,undefined,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;15202.13236.35&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{had_backfill,undefined,undefined,[]}}<br/> ** Reason for termination == <br/> ** {{badmatch,{error,closed}},<br/> &nbsp;&nbsp;&nbsp;&nbsp;[{ebucketmigrator_srv,process_upstream,2},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ebucketmigrator_srv,process_data,3},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ebucketmigrator_srv,process_data,4},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ebucketmigrator_srv,handle_info,2},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}<br/> <br/> <br/> Here are messages from this process. You can see that for some reason we see no open checkpoint on source node (current master) _and_ most weirdly we see backfill for some totally unrelated vbuckets. That may be reason why destination closes socket.<br/> <br/> [ns_server:info,2012-08-30T13:56:20.574,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:470]Setting {&quot;10.3.121.16&quot;,11209} vbucket 30 to state replica<br/> [views:debug,2012-08-30T13:56:20.598,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:452]Got set_vbucket event for saslbucket/27. Updated state: replica<br/> [views:debug,2012-08-30T13:56:20.598,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:452]Got set_vbucket event for saslbucket/27. Updated state: replica<br/> [rebalance:info,2012-08-30T13:56:20.614,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:485]Some vbuckets were not yet ready to replicate from:<br/> [30]<br/> <br/> [ns_server:debug,2012-08-30T13:56:20.614,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:kill_tapname:932]killing tap named: replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;<br/> [rebalance:info,2012-08-30T13:56:20.620,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:522]Starting tap stream:<br/> [{vbuckets,[]},<br/> &nbsp;{checkpoints,[]},<br/> &nbsp;{name,&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;&gt;&gt;},<br/> &nbsp;{takeover,false}]<br/> {{&quot;10.3.121.25&quot;,11209},<br/> &nbsp;{&quot;10.3.121.16&quot;,11209},<br/> &nbsp;[{vbuckets,[30]},<br/> &nbsp;&nbsp;{takeover,false},<br/> &nbsp;&nbsp;{suffix,&quot;building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;},<br/> &nbsp;&nbsp;{username,&quot;saslbucket&quot;},<br/> &nbsp;&nbsp;{password,&quot;password&quot;}]}<br/> [rebalance:debug,2012-08-30T13:56:20.623,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:init:555]upstream_sender pid: &lt;0.13244.35&gt;<br/> [rebalance:info,2012-08-30T13:56:20.665,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 18<br/> [rebalance:info,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 24<br/> [rebalance:info,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 25<br/> [rebalance:error,2012-08-30T13:56:20.666,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.13236.35&gt;:ebucketmigrator_srv:confirm_sent_messages:674]Got error while trying to send close confirmation: {error,enotconn}<br/> Chiyoung, I think this probably happened because for some reason we actually passed empty vbucket filter to tap start command. If that implies that ep-engine is going to send us _all_ vbuckets rather than nothing, please assign to me so that we can fix on ns_server side. Yes, if the vbucket filter is empty, it means all the vbuckets. As you probably know, this behavior exists from 1.6.x.<br/> <br/> I&#39;d like to change this definition in ep-engine, but it might require some changes in the TAP client applications that our customers developed for their own internal purpose.<br/> <br/> If it doesn&#39;t require much work in the ns-server, please go ahead and fix it in the ns-server side. Otherwise, please reassign it to me and I will then work with Steve&#39;s team to see what would be the impact on our existing customers.<br/> Liang will take a look at why we thought vbucket 30 wasn&#39;t ready to replicate from master. I&#39;ll create separate bug for handling empty ready vbuckets set in ebucketmigrator The first incident of &quot;Some vbuckets were not ready to replication from [30]&quot; on subsequent {error, enotconn} happened at time 2012-08-30T13:47:33.576 on node 16. <br/> <br/> [rebalance:info,2012-08-30T13:47:33.576,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.8085.35&gt;:ebucketmigrator_srv:init:485]Some vbuckets were not yet ready to replicate from: <br/> [30] <br/> <br/> On source node 25, the connection for replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; was closed by force, porbably due to the error condition on node 16. <br/> <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579731 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 84 not found for TAP cursor. Skip it... <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579736 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 85 not found for TAP cursor. Skip it... <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579740 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 86 not found for TAP cursor. Skip it... <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579745 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 87 not found for TAP cursor. Skip it... <br/> <br/> [ns_server:warn,2012-08-30T13:47:33.678,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:174]Dropped 2440 log lines from memcached <br/> [rebalance:info,2012-08-30T13:47:33.733,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.28940.34&gt;:ebucketmigrator_srv:do_confirm_sent_messages:650]Got close ack! <br/> <br/> [rebalance:info,2012-08-30T13:47:33.857,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.28930.34&gt;:ebucketmigrator_srv:do_confirm_sent_messages:650]Got close ack! <br/> <br/> [ns_server:info,2012-08-30T13:47:33.903,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.703424 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Connection is close by force. <br/> <br/> Interesting thing was that there was &quot;no_db_file&quot; exception exit with couch_set_view_group right before this incident on source node 25. vbucket 30 was part of add_partitions[]. <br/> <br/> [error_logger:error,2012-08-30T13:47:33.518,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.846.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{{badmatch,{not_found,no_db_file}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_set,&#39;-handle_call/3-fun-1-&#39;,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_db_set,handle_call,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.6008.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{add_partitions, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[30,31,32,52,53,54,55,56,57,58,59,74,75,76,77, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;200,201,202,203,204,211,294,295,296,297,303, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;304,305,306,307,308,358,359,360,368,369,370, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;371,372,373,374,375,376,377,378,393,394,455, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;456,469,470,471,472,473,474,477,478,479,480, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;492,493,494,570,600,601,602,608,609,610,611, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;612,616,617,618,619,620,621,622,627,628,629, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;630,631,638,639,640,643,666,667,668,669,670, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;671,680,681,682,683,684,685,686,687,688,689, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;690,700,701,702,703,723,724,725,726,727,729, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;730,731,732,733,734,735,736,737,738,739,740, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;741,742,743,744,745,762,763,764,765,766,767, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;768,769,797,798,799,800,801,839,840,841,859, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;860,861,871,872,873,874,899,900,901,902,903, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;921,936,937,938,955,980,981,982,983,1007,12,47, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;48,49,50,51,88,157,158,159,160,301,302,356,357, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;454]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.845.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.6008.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{badmatch,{not_found,no_db_file}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_set,&#39;-handle_call/3-fun-1-&#39;,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_db_set,handle_call,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.223.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 10946 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 50046050 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [ns_server:error,2012-08-30T13:47:33.525,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&#39;capi_set_view_manager-saslbucket&#39;:capi_set_view_manager:do_apply_vbucket_states:133]Failed to apply index states for the following ddocs: <br/> [{&lt;&lt;&quot;_design/d11&quot;&gt;&gt;, <br/> &nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{{{badmatch,{not_found,no_db_file}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_set,&#39;-handle_call/3-fun-1-&#39;,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_db_set,handle_call,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.6008.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{add_partitions, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[30,31,32,52,53,54,55,56,57,58,59,74,75,76,77,200,201,202, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;203,204,211,294,295,296,297,303,304,305,306,307,308,358,359, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;360,368,369,370,371,372,373,374,375,376,377,378,393,394,455, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;456,469,470,471,472,473,474,477,478,479,480,492,493,494,570, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;600,601,602,608,609,610,611,612,616,617,618,619,620,621,622, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;627,628,629,630,631,638,639,640,643,666,667,668,669,670,671, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;680,681,682,683,684,685,686,687,688,689,690,700,701,702,703, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;723,724,725,726,727,729,730,731,732,733,734,735,736,737,738, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;739,740,741,742,743,744,745,762,763,764,765,766,767,768,769, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;797,798,799,800,801,839,840,841,859,860,861,871,872,873,874, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;899,900,901,902,903,921,936,937,938,955,980,981,982,983, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1007,12,47,48,49,50,51,88,157,158,159,160,301,302,356,357, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;454]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.846.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{set_state, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[30,31,32,47,48,49,50,51,52,53,54,55,56,57,58,59,74,75,76,77, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;157,158,159,160,200,201,202,203,204,211,294,295,296,297,301, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;302,303,304,305,306,307,308,356,357,358,359,360,368,369,370, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;371,372,373,374,375,376,377,378,393,394,454,455,456,469,470, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;471,472,473,474,477,478,479,480,492,493,494,570,600,601,602, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;608,609,610,611,612,616,617,618,619,620,621,622,627,628,629, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;630,631,638,639,640,643,666,667,668,669,670,671,680,681,682, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;683,684,685,686,687,688,689,690,700,701,702,703,723,724,725, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;726,727,729,730,731,732,733,734,735,736,737,738,739,740,741, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;742,743,744,745,762,763,764,765,766,767,768,769,797,798,799, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;800,801,839,840,841,859,860,861,871,872,873,874,899,900,901, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;902,903,921,936,937,938,955,980,981,982,983,1007], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;\fX&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,17,18,19,20,21,22,23, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;24,25,26,27,28,29,33,34,35,36,37,38,39,40,41,42,43,44,45,46, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;60,61,62,63,64,65,66,67,68,69,70,71,72,73,78,79,80,81,82,83, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;84,85,86,87,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;104,105,106,107,108,109,110,111,112,113,114,115,116,117,118, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;119,120,121,122,123,124,125,126,127,128,129,130,131,132,133, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;134,135,136,137,138,139,140,141,142,143,144,145,146,147,148, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;149,150,151,152,153,154,155,156,161,162,163,164,165,166,167, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;168,169,170,171,172,173,174,175,176,177,178,179,180,181,182, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;183,184,185,186,187,188,189,190,191,192,193,194,195,196,197, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;198,199,205,206,207,208,209,210,212,213,214,215,216,217,218, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;219,220,221,222,223,224,225,226,227,228,229,230,231,232,233, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;234,235,236,237,238,239,240,241,242,243,244,245,246,247,248, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;249,250,251,252,253,254,255,256,257,258,259,260,261,262,263, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;264,265,266,267,268,269,270,271,272,273,274,275,276,277,278, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;279,280,281,282,283,284,285,286,287,288,289,290,291,292,293, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;298,299,300,309,310,311,312,313,314,315,316,317,318,319,320, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;321,322,323,324,325,326,327,328,329,330,331,332,333,334,335, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;336,337,338,339,340,341,342,343,344,345,346,347,348,349,350, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;351,352,353,354,355,361,362,363,364,365,366,367,379,380,381, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;382,383,384,385,386,387,388,389,390,391,392,395,396,397,398, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;399,400,401,402,403,404,405,406,407,408,409,410,411,412,413, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;414,415,416,417,418,419,420,421,422,423,424,425,426,427,428, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;429,430,431,432,433,434,435,436,437,438,439,440,441,442,443, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;444,445,446,447,448,449,450,451,452,453,457,458,459,460,461, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;462,463,464,465,466,467,468,475,476,481,482,483,484,485,486, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;487,488,489,490,491,495,496,497,498,499,500,501,502,503,504, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;505,506,507,508,509,510,511,512,513,514,515,516,517,518,519, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;520,521,522,523,524,525,526,527,528,529,530,531,532,533,534, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;535,536,537,538,539,540,541,542,543,544,545,546,547,548,549, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;550,551,552,553,554,555,556,557,558,559,560,561,562,563,564, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;565,566,567,568,569,571,572,573,574,575,576,577,578,579,580, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;581,582,583,584,585,586,587,588,589,590,591,592,593,594,595, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;596,597,598,599,603,604,605,606,607,613,614,615,623,624,625, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;626,632,633,634,635,636,637,641,642,644,645,646,647,648,649, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;650,651,652,653,654,655,656,657,658,659,660,661,662,663,664, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;665,672,673,674,675,676,677,678,679,691,692,693,694,695,696, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;697,698,699,704,705,706,707,708,709,710,711,712,713,714,715, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;716,717,718,719,720,721,722,728,746,747,748,749,750,751,752, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;753,754,755,756,757,758,759,760,761,770,771,772,773,774,775, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;776,777,778,779,780,781,782,783,784,785,786,787,788,789,790, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;791,792,793,794,795,796,802,803,804,805,806,807,808,809,810, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;811,812,813,814,815,816,817,818,819,820,821,822,823,824,825, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;826,827,828,829,830,831,832,833,834,835,836,837,838,842,843, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;844,845,846,847,848,849,850,851,852,853,854,855,856,857,858, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;862,863,864,865,866,867,868,869,870,875,876,877,878,879,880, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;881,882,883,884,885,886,887,888,889,890,891,892,893,894,895, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;896,897,898,904,905,906,907,908,909,910,911,912,913,914,915, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;916,917,918,919,920,922,923,924,925,926,927,928,929,930,931, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;932,933,934,935,939,940,941,942,943,944,945,946,947,948,949, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;950,951,952,953,954,956,957,958,959,960,961,962,963,964,965, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;966,967,968,969,970,971,972,973,974,975,976,977,978,979,984, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;985,986,987,988,989,990,991,992,993,994,995,996,997,998,999, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1000,1001,1002,1003,1004,1005,1006,1008,1009,1010,1011,1012, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}}}] <br/> Search for vb_30 backward in node 25 log, I saw that vb_30 has &quot;persisted_checkpoint_id&quot; is &lt;&lt;&quot;0&quot;&gt;&gt;. But, this seems too far back in time. <br/> <br/> [ns_server:info,2012-08-29T19:21:51.992,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.31883.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;100&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> &nbsp;.... <br/> <br/> {&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> &nbsp; <br/> &nbsp;&nbsp; <br/> &nbsp; Liang, <br/> <br/> &gt;On source node 25, the connection for replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; was closed by force, porbably due to the error condition on node 16. <br/> <br/> &gt;memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579731 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 84 not found for TAP cursor. Skip it... <br/> &gt;memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579736 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 85 not found for TAP cursor. Skip it... <br/> &gt;memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579740 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 86 not found for TAP cursor. Skip it... <br/> &gt;memcached&lt;0.654.0&gt;: Thu Aug 30 20:47:33.579745 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - VBucket 87 not found for TAP cursor. Skip it... <br/> <br/> <br/> This was caused by the empty vbucket filter for the TAP producer &quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;. If the ns-server passed the empty vbucket filter to a given TAP producer, the TAP producer assumes all vbuckets and iterates them one by one. <br/> <br/> The expected vbucket filter for the above producer should have vbucket 30 only. <br/> <br/> {&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> <br/> <br/> These checkpoint stats are interesting. We don&#39;t expect the active vbucket to have 0 as the open checkpoint id. We reset the open checkpoint id to 0 for a replica vbucket right before it receives backfill items from the producer. After backfill is completed, the replica vbucket receives the positive open checkpoint id from the producer. Then, we change the vbucket state from replica to active iff this is for vbucket takeover. <br/> actually, node_16 and node_22 hit the &quot;empty vbucket filter&quot; problem and {error,enotconn} even earlier. The first incident is right after config change in vbucket map w.r.t vbucket 30. <br/> <br/> [rebalance:info,2012-08-29T19:21:45.693,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.3763.3&gt;:ebucketmigrator_srv:init:485]Some vbuckets were not yet ready to replicate from: <br/> [30] <br/> <br/> Some sequence of what-is-happening in time order in node-25 log, <br/> <br/> 1, When replica chain was {node_13,node_16, node_22} <br/> <br/> [ns_server:debug,2012-08-29T19:11:10.271,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_config_log:ns_config_log:log_common:111]config change: <br/> buckets -&gt; <br/> [{configs,[{&quot;default&quot;, <br/> ........... <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{30,[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{31,[], <br/> <br/> The diag had vb_30:persisted_checkpoint_id = &lt;&lt;&quot;0&quot;&gt;&gt; <br/> <br/> [ns_server:info,2012-08-29T19:12:00.001,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25668.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;100&quot;&gt;&gt;}, <br/> ........ <br/> &nbsp;{&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> <br/> <br/> Before the next config change w.r.t vbucket 30, I saw that replication_building_30 to node_13,node_16,and node_22 completed and closed. <br/> <br/> 2,, Now, some config change that has a new replica chain for vb_30 [node_25, node_16, node_22]. This is before node_22 received not-ready-to-replicate-from [30] error. <br/> <br/> [ns_server:debug,2012-08-29T19:21:44.660,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_config_log:ns_config_log:log_common:111]config change: <br/> buckets -&gt; <br/> [{configs,[{&quot;default&quot;, <br/> ............. <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{30,[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;]}, <br/> <br/> 3, Node_16 (and node_22) both hit vbucket-not-ready [30] shortly after. <br/> <br/> [rebalance:info,2012-08-29T19:21:45.693,<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>:&lt;0.3763.3&gt;:ebucketmigrator_srv:init:485]Some vbuckets were not yet ready to replicate from: <br/> [30] <br/> <br/> <br/> 4, At source node_25, due to receiving empty vbucket filter, it was trying to send all vbuckets .... <br/> <br/> [ns_server:info,2012-08-29T19:21:45.923,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:21:45.709565 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - VBucket 1 not found for TAP cursor. Skip it... <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:21:45.709583 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - VBucket 2 not found for TAP cursor. Skip it... <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:21:45.709590 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - VBucket 3 not found for TAP cursor. Skip it... <br/> <br/> 5, At destination node_16 and node_22, {error, enotconn} happened thereafter. <br/> <br/> 6, Interest log is that vb_30 has persisted_checkpoint_id = &quot;99&quot; as shown by diag <br/> <br/> [ns_server:info,2012-08-29T19:21:51.992,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.31883.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;......... <br/> &nbsp;{&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.26'>replication_ns_1@10.3.121.26</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;100&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;100&quot;&gt;&gt;}, <br/> <br/> 7, Confusing thing is - the next diag shortly after that show &quot;0&quot; for vb_30 persisted_checkpoint_id. <br/> <br/> [ns_server:info,2012-08-29T19:21:51.992,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.31883.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> ............. <br/> {&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> Some additional findings of how vbucket 30 persisted_checkpoint_id became &quot;0&quot; on node_25. <br/> <br/> 1, At timestamp 08-29T19:11:59, vbucket 30 had persisted_checkpoint_id &quot;98&quot;. <br/> <br/> [ns_server:info,2012-08-29T19:11:59.841,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25668.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:default: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;97&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;98&quot;&gt;&gt;}, <br/> .............. <br/> &nbsp;{&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;98&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;63&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;64&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;98&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> <br/> 2, Some couchdb set view log shows vbucket 30 in cleanup partition, and then in pending active partition. Apparently, something happened here and memcached cleared checkpoint id for vbucket 30. <br/> <br/> [couchdb:info,2012-08-29T19:11:59.918,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.846.0&gt;:couch_log:info:39]Set view `saslbucket`, main group `_design/d11`, partition states updated <br/> active partitions before: [31,32,52,53,54,55,56,57,58,59,74,75,76,77,200,201,202,203,204,211,294,295,296,297,303,304,305,306,307,308,358,359,360,368,369,370,371,372,373,374,375,376,377,378,393,394,469,470,471,472,473,474,492,493,494,570,600,601,602,608,609,610,611,612,616,617,618,619,620,621,622,627,628,629,630,631,638,639,640,666,667,668,669,670,671,683,684,685,686,687,688,689,690,700,701,702,703,723,724,725,726,727,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,762,763,764,765,766,767,768,769,839,840,841,859,860,861,871,872,873,874,936,937,938,955,980,981,982,983] <br/> active partitions after: [31,32,52,53,54,55,56,57,58,59,74,75,76,77,200,201,202,203,204,211,294,295,296,297,303,304,305,306,307,308,358,359,360,368,369,370,371,372,373,374,375,376,377,378,393,394,469,470,471,472,473,474,492,493,494,570,600,601,602,608,609,610,611,612,616,617,618,619,620,621,622,627,628,629,630,631,638,639,640,666,667,668,669,670,671,683,684,685,686,687,688,689,690,700,701,702,703,723,724,725,726,727,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,762,763,764,765,766,767,768,769,839,840,841,859,860,861,871,872,873,874,936,937,938,955,980,981,982,983] <br/> passive partitions before: [47,48,49,50,51,301,302,356,357,479,480,899,900,901,902,903,921,1007] <br/> passive partitions after: [47,48,49,50,51,301,302,356,357,479,480,899,900,901,902,903,921,1007] <br/> cleanup partitions before: [30,189] <br/> cleanup partitions after: [30,189] <br/> unindexable partitions: [] <br/> replica partitions before: [0,18,24,25,26,27,47,48,49,50,51,60,61,62,88,89,99,100,101,102,121,122,123,124,125,126,133,134,135,136,137,138,157,158,159,160,198,199,225,226,227,248,249,250,251,252,253,276,277,278,279,280,292,293,298,299,300,301,302,356,357,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,429,430,431,432,433,454,455,456,457,458,459,460,461,462,477,478,479,480,488,489,490,491,495,496,497,498,509,510,511,513,533,534,535,554,555,565,566,567,568,569,582,583,584,585,586,587,595,596,597,598,599,623,624,625,626,641,642,643,649,650,651,652,653,654,655,656,657,658,659,672,673,674,675,676,677,678,679,680,681,682,697,698,699,710,711,712,718,719,720,721,722,728,757,758,759,760,761,773,774,775,776,777,778,779,785,786,787,788,789,790,791,792,797,798,799,800,801,819,820,821,822,823,824,825,826,827,832,833,834,835,836,837,838,842,877,878,879,881,882,883,884,886,893,894,895,896,897,899,900,901,902,903,910,911,912,913,914,921,962,963,964,967,968,969,970,971,1001,1002,1003,1007] <br/> replica partitions after: [0,18,24,25,26,27,47,48,49,50,51,60,61,62,88,89,99,100,101,102,121,122,123,124,125,126,133,134,135,136,137,138,157,158,159,160,198,199,225,226,227,248,249,250,251,252,253,276,277,278,279,280,292,293,298,299,300,301,302,356,357,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,429,430,431,432,433,454,455,456,457,458,459,460,461,462,477,478,479,480,488,489,490,491,495,496,497,498,509,510,511,513,533,534,535,554,555,565,566,567,568,569,582,583,584,585,586,587,595,596,597,598,599,623,624,625,626,641,642,643,649,650,651,652,653,654,655,656,657,658,659,672,673,674,675,676,677,678,679,680,681,682,697,698,699,710,711,712,718,719,720,721,722,728,757,758,759,760,761,773,774,775,776,777,778,779,785,786,787,788,789,790,791,792,797,798,799,800,801,819,820,821,822,823,824,825,826,827,832,833,834,835,836,837,838,842,877,878,879,881,882,883,884,886,893,894,895,896,897,899,900,901,902,903,910,911,912,913,914,921,962,963,964,967,968,969,970,971,1001,1002,1003,1007] <br/> replicas on transfer before: [47,48,49,50,51,301,302,356,357,479,480,899,900,901,902,903,921,1007] <br/> replicas on transfer after: [47,48,49,50,51,301,302,356,357,479,480,899,900,901,902,903,921,1007] <br/> pending transition before: <br/> &nbsp;&nbsp;active: [30] <br/> &nbsp;&nbsp;passive: [189] <br/> pending transition after: <br/> &nbsp;&nbsp;active: [30] <br/> &nbsp;&nbsp;passive: [] <br/> <br/> 3, Right after that, replication_building_30_* tap queues were cleared, from node_25 to node_16, node_13, and node_22. Note that node_13 left cluster at timestamp 08-29T19:21:42. <br/> <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.798390 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Clear the tap queues by force <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.798626 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Clear the tap queues by force <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.798687 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Clear the tap queues by force <br/> <br/> 4, Now, diag showed that vbucket 30 persisted_checkpoint_id became &quot;0&quot;. <br/> <br/> [ns_server:info,2012-08-29T19:12:00.001,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25668.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;100&quot;&gt;&gt;}, <br/> &nbsp;............... <br/> {&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> &nbsp; <br/> Continue with the log messages I put in the previous comment. Node_13 left the cluster at timestamp 08-29T19:21:42, and then config change at 08-29T19:21:44. Hopefully, we can get some idea now. Going a bit further back from last comment, more interesting findings - <br/> <br/> 1, All 3 replication_building_30_* tap connections were healthy at timestamp 08-29T19:11:41 <br/> <br/> [ns_server:info,2012-08-29T19:11:41.305,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25209.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> <br/> {&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;98&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;3&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;98&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> <br/> 2, Replication_building_30_&#39;node_13&#39; closed because backfill of bbucket 30 completed. But, I didn&#39;t see backfill complete for node_16 and node_22. Note, vbucket 30 showe dup in replication_ns_1@node_13 vbucket filter laster. <br/> <br/> [ns_server:info,2012-08-29T19:11:53.989,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:53.786917 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Backfill is completed with VBuckets 30, <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:53.787071 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Sending TAP_OPAQUE with command &quot;close_backfill&quot; and vbucket 30 <br/> <br/> 3, For some reason, replication_building_30_ to node_22 and node_16 were closed by force, at time 08-29T19:21:59. It seems that backfill for vbucket 30 were not complete on both destination nodes. <br/> <br/> [ns_server:info,2012-08-29T19:11:59.240,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.038795 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - disconnected, keep alive for 300 seconds <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.043319 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - disconnected, keep alive for 300 seconds <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.060846 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Connection is closed by force. <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.061353 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Connection is closed by force. <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.061713 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Connection is closed by force. <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:59.084733 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - disconnected <br/> <br/> 4, Janitor_agent does &quot;undoing temp vbucket state&quot;, and note that vbucket 30 was in active partition as shown below. But, it was moved to cleanup partition which caused persisted_checkpoint_id changes. See log msg in above comment. <br/> <br/> [ns_server:info,2012-08-29T19:11:59.537,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&#39;janitor_agent-saslbucket&#39;:janitor_agent:handle_info:640]Undoing temporary vbucket states caused by rebalance <br/> [ns_server:info,2012-08-29T19:11:59.550,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25668.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:126]logging tap &amp; checkpoint stats <br/> [views:info,2012-08-29T19:11:59.545,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&#39;capi_ddoc_replication_srv-saslbucket&#39;:capi_set_view_manager:apply_index_states:351] <br/> Calling couch_set_view:set_partition_states([&lt;&lt;&quot;saslbucket&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;_design/d11&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[30,31,32,47,48,49,50,51,52,53, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;54,55,56,57,58,59,74,75,76,77, <br/> <br/> I think the dots are connected here ...... Sum it up - <br/> <br/> 1, There were replication_building for vbucket 30 from node_25 to node_13, node_16, and node_22; <br/> 2, Backfill vbucket 30 to node_13 completed; <br/> 3, Rebalance somehow was stopped before it even reached takeover phase; <br/> 4, At this point, node_25 diag had vbucket 30 persisted_checkpoint_id = &quot;98&quot; and open_checkpoint_id = &quot;99&quot;; <br/> 5, Some couch_set_view to set partition state, with cleanup_partitions = {30}; <br/> 6, Now, node_25 diag had vbucket 30 persisted_checkpoint_id became &quot;0&quot;; <br/> 7, Node_13 left the cluster, and this resulted in some config change which cause replication building from node_25 to node_16 and node_22 again; <br/> 8, Vbucket 30 not ready on node_25 (=&quot;0&quot;) to replicate. <br/> <br/> Talked to Chiyoung. For a vbucket checkpoint id to become 0, there must be 1) vbucket deletion or 2) initial vbucket stream for backfill. However, initial vbucket stream to node_25 does not make sense because it is the current master. Need to understand whether there was vbucket deletion by couch_set_view. <br/> <br/> Hi Filipe, <br/> <br/> I tried to understand what couch_set_view does for cleanup_partitions = {30} but the erlang code there totally puzzles me :). So, I want to ask you what exactly couch_set_view_group do with a vbucket (30) in cleanup_partitions[]. Any vbucket deletion? Please see my comments above. Feel free to assign it back to me. <br/> <br/> Thanks, <br/> Liang Hi Liang,<br/> <br/> Nop, the indexes only read from databases. They never write to databases nor delete them.<br/> <br/> Marking a vbucket (database) for cleanup in an index means to delete all the indexed data originated from that database from the index (doesn&#39;t touch database files, nor deletes them). Hi Liang, <br/> Any news/status on this one? <br/> Thanks, <br/> Steve I&#39;d narrowed down to where vbucket 30 open checkpoint id became 0 from 99 in a tiny window. There is no additional traces in ns_server log for me to dig. There is no memcached log or ns_server log showing any vbucket state change on node_25. Dunno what else I could do with this. Aliaksey/Liang, <br/> <br/> given that we did not see this rebalance failure in system testing do you think this was a one-time issue that is very rare to occur ? It is possible one of recent ep-engine fixes closed this. Hard to say. Still we&#39;d like to understand. Rareness is IMHO not a good argument. I.e. there&#39;s possibility of massive data loss at stake here. Entire master vbucket was deleted somehow. I.e. there&#39;s possibility of massive data loss at stake here. Entire master vbucket was deleted somehow. Here&#39;re my observations. Node 23 is the master node of the cluster. At some <br/> point during rebalance it decides to build replicas of vbucket 30 on nodes 13, <br/> 16 and 22. Current owner of this vbucket is node 25: <br/> <br/> [rebalance:debug,2012-08-29T19:11:20.428,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.31805.466&gt;:ns_single_vbucket_mover:mover_inner_old_style:190]child replicas builder for vbucket 30 is &lt;0.31941.466&gt; <br/> [ns_server:debug,2012-08-29T19:11:20.436,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.31941.466&gt;:ns_replicas_builder_utils:spawn_replica_builder:86]Replica building ebucketmigrator for vbucket 30 into &#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; is &lt;15197.23780.3&gt; <br/> [ns_server:debug,2012-08-29T19:11:20.447,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.31941.466&gt;:ns_replicas_builder_utils:spawn_replica_builder:86]Replica building ebucketmigrator for vbucket 30 into &#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; is &lt;15202.29015.2&gt; <br/> [ns_server:debug,2012-08-29T19:11:20.453,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.31941.466&gt;:ns_replicas_builder_utils:spawn_replica_builder:86]Replica building ebucketmigrator for vbucket 30 into &#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; is &lt;15205.9270.2&gt; <br/> <br/> <br/> Destination nodes start ebucketmigrators for replica building. It means that <br/> at this particular moment of time node 25&#39;s vbucket 30 still has non-zero <br/> checkpoint id: <br/> <br/> [ns_server:debug,2012-08-29T19:11:20.487,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.23780.3&gt;:ebucketmigrator_srv:kill_tapname:932]killing tap named: replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; <br/> [rebalance:info,2012-08-29T19:11:20.507,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.23780.3&gt;:ebucketmigrator_srv:init:522]Starting tap stream: <br/> [{vbuckets,[30]}, <br/> &nbsp;{checkpoints,[{30,0}]}, <br/> &nbsp;{name,&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;&quot;&gt;&gt;}, <br/> &nbsp;{takeover,false}] <br/> {{&quot;10.3.121.25&quot;,11209}, <br/> &nbsp;{&quot;10.3.121.13&quot;,11209}, <br/> &nbsp;[{vbuckets,[30]}, <br/> &nbsp;&nbsp;{takeover,false}, <br/> &nbsp;&nbsp;{suffix,&quot;building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;&quot;}, <br/> &nbsp;&nbsp;{username,&quot;saslbucket&quot;}, <br/> &nbsp;&nbsp;{password,&quot;password&quot;}]} <br/> [rebalance:debug,2012-08-29T19:11:20.528,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.23780.3&gt;:ebucketmigrator_srv:init:555]upstream_sender pid: &lt;0.23783.3&gt; <br/> [rebalance:info,2012-08-29T19:11:20.543,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.23780.3&gt;:ebucketmigrator_srv:process_upstream:880]Initial stream for vbucket 30 <br/> <br/> <br/> On node 25 we see that memcached accordingly schedules backfills to 13, 16 and 22: <br/> <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.515187 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Schedule the backfill for vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.515253 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.515279 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.515289 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.545809 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Schedule the backfill for vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.545856 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.545872 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.545882 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.551288 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Schedule the backfill for vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.551456 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.551477 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.551487 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 30 <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:20.559238 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Suspend for 5.00 secs <br/> <br/> <br/> And sooner or later those backfills complete: <br/> <br/> [ns_server:info,2012-08-29T19:11:23.286,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:23.086272 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Backfill is completed with VBuckets 30, <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:23.086351 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39; - Sending TAP_OPAQUE with command &quot;close_backfill&quot; and vbucket 30 <br/> [ns_server:info,2012-08-29T19:11:24.864,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:24.663901 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Backfill is completed with VBuckets 30, <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:24.670059 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; - Sending TAP_OPAQUE with command &quot;close_backfill&quot; and vbucket 30 <br/> [ns_server:info,2012-08-29T19:11:25.887,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:25.686315 3: TAP (Producer) eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a> - Suspend for 5.00 secs <br/> <br/> <br/> [ns_server:info,2012-08-29T19:11:53.989,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:53.786917 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Backfill is completed with VBuckets 30, <br/> memcached&lt;0.654.0&gt;: Thu Aug 30 02:11:53.787071 3: TAP (Producer) eq_tapq:replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - Sending TAP_OPAQUE with command &quot;close_backfill&quot; and vbucket 30 <br/> <br/> <br/> Then rebalance gets stopped by user: <br/> <br/> [ns_server:info,2012-08-29T19:11:59.054,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.31941.466&gt;:ns_replicas_builder_utils:kill_a_bunch_of_tap_names:59]Killed the following tap names on &#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39;: [&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_30_&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;&quot;&gt;&gt;] <br/> <br/> [user:info,2012-08-29T19:11:59.529,<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>:&lt;0.6719.464&gt;:ns_orchestrator:handle_info:289]Rebalance stopped by user. <br/> <br/> <br/> As part of the code that handles rebalance completion (for any reason) we <br/> immediately log all tap and checkpoint stats on all the nodes. In these stats <br/> on 25 we can see that vbucket 30 already has checkpoint id of zero: <br/> <br/> [ns_server:info,2012-08-29T19:12:00.001,<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>:&lt;0.25668.2&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1007:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;99&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_1007:eq_tapq:<a href='mailto:replication_ns_1@10.3.121.13'>replication_ns_1@10.3.121.13</a>:cursor_checkpoint_id&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;..... <br/> &nbsp;{&lt;&lt;&quot;vb_30:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;}, <br/> &nbsp;{&lt;&lt;&quot;vb_30:state&quot;&gt;&gt;,&lt;&lt;&quot;active&quot;&gt;&gt;}, <br/> <br/> <br/> I could not find anything abnormal from ns_server perspective during that <br/> period of time. So probably it&#39;s something wrong in ep_engine. Apparently more <br/> logs from ep_engine would help here. Particularly it would be good if vbucket <br/> resets were logged. <br/> Chiyoung, <br/> <br/> do you require more information from QE to proceed ? in that case please assign te bug back to Tony. <br/> <br/> also this issue is not reproduced easily in system testing so not sure if Tony can reproduce and collect the new logs so in that case would you be okay with deferring this bug for now ? This issue looks too strange to me. When the rebalance got stopped by the user, then the current master should still hold the active vbucket 30 with non-zero open checkpoint id. The open checkpoint id is reset to zero iff its vbucket get reset by receiving the backfill stream from the master or when it&#39;s deleted and recreated with non-active state. <br/> <br/> I will add more logs to ep-engine, but don&#39;t see anything suspicious in ep-engine at this time. We will see if this issue happens again in 2.0 full regressions. I&#39;m fine with moving this to post 2.0 given the fact that it&#39;s not easily reproducible. Integrated in github-ep-engine-2-0 #454 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/454/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/454/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6494" title="[longevity] rebalance fails with error &quot;Got error while trying to send close confirmation: {error,enotconn} during rebalance&quot; when adding a failed over node back to the cluster and failing over another node"><strike>MB-6494</strike></a> Add more logs to vbucket reset and deletion functions (Revision 96d50b2038777e25f671475ef8fad76ba3a76683)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * src/ep_engine.cc<br/> We spent lots of time on debugging this issue, but not able to reproduce it. We added more logs to the ep-engine. Let&#39;s open the new bug when we hit the this issue again. Date of First Response Thu, 30 Aug 2012 16:38:16 -0500 Rank 4114 Time In Status