http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-4500 Couchbase Server Install couchbase server 2.0.0r-266 on 3 nodes at ec2. <br/> Use mcsoda to load 200000 items to cluster. <br/> Create a view name one and do some query. <br/> Shutdown one node (A) and failover <br/> Check reduce count on view one. Ok <br/> Reinstall couchbase server 2.0.0r-266 on node A and add it back to cluster. <br/> Rebalance. Ok <br/> Check reduce count on view one. Ok <br/> Shutdown node B and failover. <br/> Check reduce count on cluster. Ok <br/> Reinstall couchbase server 2.0.0r-266 on node B and add it back to cluster. <br/> Rebalance Ok. <br/> Check reduce count on full cluster. Failed <br/> Restart couchbase server on 3 nodes of cluster. <br/> See reduce count is different on each node <br/> <br/> centos 5.4 64bit on ec2 MB-4500

reduce map show different count number at each node

Bug Blocker Closed Fixed Public Aliaksey Artamonau Thuan Nguyen 2.0-DP3-release-notes 2.0-dev-preview-4-release-notes Thu, 1 Dec 2011 21:36:49 -0600 Wed, 9 Jan 2013 22:59:12 -0600 Wed, 18 Jan 2012 20:04:07 -0600 2.0-developer-preview-3 2.0-developer-preview-4 view-engine 0 1 So greping each log for the last occurrence of &quot;Set view `default`, group `_design/dev_one`, partition states updated&quot;, I can see that node 187 has the index without any active partitions defined. <br/> <br/> log187: <br/> <br/> [couchdb:info] [2011-12-01 18:28:54] [<a href='mailto:ns_1@10.98.186.187'>ns_1@10.98.186.187</a>:&lt;0.19609.9&gt;:couch_log:info:39] Set view `default`, group `_design/dev_one`, partition states updated <br/> abitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, abitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> pbitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000111111111111111111111111111111111111111111, pbitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> cbitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000, cbitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000011111111111111111111111111111111111111111111000000000000000000000000000000000000000000111111111111111111111111111111111111111111 <br/> <br/> log153: <br/> <br/> [couchdb:info] [2011-12-01 18:28:54] [<a href='mailto:ns_1@10.124.193.153'>ns_1@10.124.193.153</a>:&lt;0.32606.0&gt;:couch_log:info:39] Set view `default`, group `_design/dev_one`, partition states updated <br/> abitmask before 1111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000001111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000, abitmask after 1111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> pbitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, pbitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> cbitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, cbitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> <br/> log206: <br/> <br/> [couchdb:info] [2011-12-01 18:28:52] [<a href='mailto:ns_1@10.90.182.206'>ns_1@10.90.182.206</a>:&lt;0.31550.0&gt;:couch_log:info:39] Set view `default`, group `_design/dev_one`, partition states updated <br/> abitmask before 1000000000000000000000000000000000000000000000000000000000000000000000000000001110000011111111111111111111111111111111111111111100000000000000000000000000000000000000000001111111111111111111111111111111111111111111000000000000000000000000000000000000000000, abitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000001110000011111111111111111111111111111111111111111100000000000000000000000000000000000000000001111111111111111111111111111111111111111111000000000000000000000000000000000000000000 <br/> pbitmask before 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, pbitmask after 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> cbitmask before 0111111111111111111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, cbitmask after 1111111111111111111111111111111111111111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 <br/> <br/> In an Erlang shell, we can see that in the whole cluster, we only have 173 active partitions instead of 256: <br/> <br/> 2&gt; N206 = couch_set_view_util:decode_bitmask(2#0000000000000000000000000000000000000000000000000000000000000000000000000000001110000011111111111111111111111111111111111111111100000000000000000000000000000000000000000001111111111111111111111111111111111111111111000000000000000000000000000000000000000000). <br/> [42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60, <br/> &nbsp;61,62,63,64,65,66,67,68,69,70|...] <br/> 3&gt; N187 = couch_set_view_util:decode_bitmask(2#0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000). <br/> [] <br/> 4&gt; N153 = couch_set_view_util:decode_bitmask(2#1111111111111111111111111111111111111111111000000000000000000000000000000000000000000000000000000000000000000000000000000000000011111111111111111111111111111111111111111110000000000000000000000000000000000000000000000000000000000000000000000000000000000000). <br/> [85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102, <br/> &nbsp;103,104,105,106,107,108,109,110,111,112,113|...] <br/> 5&gt; <br/> 5&gt; io:format(&quot;~w~n&quot;, [N206]). <br/> [42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,175,176,177] <br/> ok <br/> 6&gt; io:format(&quot;~w~n&quot;, [N153]). <br/> [85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255] <br/> ok <br/> 7&gt; <br/> 7&gt; O1 = ordsets:from_list(N153). <br/> [85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102, <br/> &nbsp;103,104,105,106,107,108,109,110,111,112,113|...] <br/> 8&gt; O2 = ordsets:union(O1, ordsets:from_list(N206)). <br/> [42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60, <br/> &nbsp;61,62,63,64,65,66,67,68,69,70|...] <br/> 9&gt; <br/> 9&gt; io:format(&quot;~p~n&quot;, [O2]). <br/> [42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66, <br/> &nbsp;67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91, <br/> &nbsp;92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112, <br/> &nbsp;113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131, <br/> &nbsp;132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150, <br/> &nbsp;151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169, <br/> &nbsp;175,176,177,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228, <br/> &nbsp;229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247, <br/> &nbsp;248,249,250,251,252,253,254,255] <br/> ok <br/> 10&gt; length(O2). <br/> 174 <br/> <br/> Is it possible ns_server missed an index state update on node 187? Is the reduce result incorrect only if node fails over , cluster gets rebalanced and node gets added? Tony, can you retest with latest UI? We think we&#39;re hitting indexing timeouts. Newer UI will indicate that. <br/> <br/> Also as part of fixing it we need to be able to specify very big timeouts so that any indexing activity can be performed. <br/> marking this resolved as Aliaksey A (standing over my desk here) believe it&#39;s fixed Date of First Response Fri, 2 Dec 2011 07:00:47 -0600 Rank 5943 Time In Status