http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6967 Couchbase Server i tried the following scenario on the latest build: <br/> 1. Create cluster of 2 nodes. load 1M items, 1K each with cbworkloadgen ./cbworkloadgen -n localhost:8091 -i10000000 -t6 -j -s1000 <br/> 2. Create 1DD, with 4 simple views, of which 2 had _count reduce function so i can count all documents on that view. <br/> 3. I built the index and confirmed that the _count views return 1M items, i.e. index was built successfully. <br/> 4. I added 2 more nodes to the cluster and rebalance. <br/> 5. During the rebalance, i refreshed the _count views. <br/> Expected behavior (with consistent views enabled by default) is that i will get the same 1M items during the reblance <br/> Observed behavior: the number kept changing, and decreasing over time to 900K, 800K.. <br/> <br/> Iryna, can you collect the neccesary logs from this cluster and post it on this bug. <br/> Cluster can be found at: <a href="http://184.169.209.178:8091/">http://184.169.209.178:8091/</a> (usual credentials) <br/> Here are all the IP addresses: <br/> <a href='mailto:ns_1@10.176.29.176'>ns_1@10.176.29.176</a> 10.176.29.176 184.169.209.178 <a href='mailto:ns_1@10.176.9.41'>ns_1@10.176.9.41</a> <br/> <a href='mailto:ns_1@10.176.9.41'>ns_1@10.176.9.41</a> 10.176.9.41 50.18.23.114 <a href='mailto:ns_1@10.176.9.41'>ns_1@10.176.9.41</a> <br/> <a href='mailto:ns_1@10.168.103.76'>ns_1@10.168.103.76</a> 10.168.103.76 204.236.154.91 <a href='mailto:ns_1@10.176.9.41'>ns_1@10.176.9.41</a> <br/> <a href='mailto:ns_1@10.176.145.104'>ns_1@10.176.145.104</a> 10.176.145.104 54.241.117.117 <a href='mailto:ns_1@10.176.9.41'>ns_1@10.176.9.41</a> <br/> <br/> <br/> I did a few things on this cluster like creating and removing indexes/nodes. so look at the logs at the end. <br/> <br/> <br/> MAC OS <br/> build 1868 <br/> <br/> &lt;manifest&gt;&lt;remote name=&quot;couchbase&quot; fetch=&quot;<a href="git://10.1.1.210/">git://10.1.1.210/</a>&quot;/&gt;&lt;remote name=&quot;membase&quot; fetch=&quot;<a href="git://10.1.1.210/">git://10.1.1.210/</a>&quot;/&gt;&lt;remote name=&quot;apache&quot; fetch=&quot;<a href="git://github.com/apache/">git://github.com/apache/</a>&quot;/&gt;&lt;remote name=&quot;erlang&quot; fetch=&quot;<a href="git://github.com/erlang/">git://github.com/erlang/</a>&quot;/&gt;&lt;default remote=&quot;couchbase&quot; revision=&quot;master&quot;/&gt;&lt;project name=&quot;tlm&quot; path=&quot;tlm&quot; revision=&quot;ab70f6d42f46621ec576889e57cb37ac2d64a84b&quot;&gt;&lt;copyfile dest=&quot;Makefile&quot; src=&quot;Makefile.top&quot;/&gt;&lt;/project&gt;&lt;project name=&quot;bucket_engine&quot; path=&quot;bucket_engine&quot; revision=&quot;70b3624abc697b7d18bf3d57f331b7674544e1e7&quot;/&gt;&lt;project name=&quot;ep-engine&quot; path=&quot;ep-engine&quot; revision=&quot;10b593cf4d97eaf062a6076878c5f8000d093ee9&quot;/&gt;&lt;project name=&quot;libconflate&quot; path=&quot;libconflate&quot; revision=&quot;2cc8eff8e77d497d9f03a30fafaecb85280535d6&quot;/&gt;&lt;project name=&quot;libmemcached&quot; path=&quot;libmemcached&quot; revision=&quot;ca739a890349ac36dc79447e37da7caa9ae819f5&quot; remote=&quot;membase&quot;/&gt;&lt;project name=&quot;libvbucket&quot; path=&quot;libvbucket&quot; revision=&quot;00d3763593c116e8e5d97aa0b646c42885727398&quot;/&gt;&lt;project name=&quot;membase-cli&quot; path=&quot;membase-cli&quot; revision=&quot;7fe4121e7e83952a4cb032e25a2cb9fca1709354&quot; remote=&quot;membase&quot;/&gt;&lt;project name=&quot;memcached&quot; path=&quot;memcached&quot; revision=&quot;06ab906e6702917c4b6b90a6b0051644719a357d&quot; remote=&quot;membase&quot;/&gt;&lt;project name=&quot;moxi&quot; path=&quot;moxi&quot; revision=&quot;52a5fa887bfff0bf719c4ee5f29634dd8707500e&quot;/&gt;&lt;project name=&quot;ns_server&quot; path=&quot;ns_server&quot; revision=&quot;82c7c95e33445edcc6665e64216983e1027756e1&quot;/&gt;&lt;project name=&quot;portsigar&quot; path=&quot;portsigar&quot; revision=&quot;1bc865e1622fb93a3fe0d1a4cdf18eb97ed9d600&quot;/&gt;&lt;project name=&quot;sigar&quot; path=&quot;sigar&quot; revision=&quot;63a3cd1b316d2d4aa6dd31ce8fc66101b983e0b0&quot;/&gt;&lt;project name=&quot;couchbase-examples&quot; path=&quot;couchbase-examples&quot; revision=&quot;21e6161a1d064979b5c6aa99cd34ccc41c9d7aca&quot;/&gt;&lt;project name=&quot;couchbase-python-client&quot; path=&quot;couchbase-python-client&quot; revision=&quot;86b398e4fbc1f2e38d356e14df0c1bb4e3d2427b&quot;/&gt;&lt;project name=&quot;couchdb&quot; path=&quot;couchdb&quot; revision=&quot;69032c6dbcd64c056265416d2c49b1a9ee06f9c3&quot;/&gt;&lt;project name=&quot;couchdbx-app&quot; path=&quot;couchdbx-app&quot; revision=&quot;76d79be79c1454cff0f878d5a88a792270ec1b17&quot;/&gt;&lt;project name=&quot;couchstore&quot; path=&quot;couchstore&quot; revision=&quot;772f66a29a81be59d8bdaaa74b3898bb44fcc7e2&quot;/&gt;&lt;project name=&quot;geocouch&quot; path=&quot;geocouch&quot; revision=&quot;b0bd742551639c52030c070e5bf9390edbb536ba&quot;/&gt;&lt;project name=&quot;mccouch&quot; path=&quot;mccouch&quot; revision=&quot;88701cc326bc3dde4ed072bb8441be83adcfb2a5&quot;/&gt;&lt;project name=&quot;testrunner&quot; path=&quot;testrunner&quot; revision=&quot;621f470d068b2d20fce7b323e5f9fd44304db76c&quot;/&gt;&lt;project name=&quot;otp&quot; path=&quot;otp&quot; revision=&quot;b6dc1a844eab061d0a7153d46e7e68296f15a504&quot; remote=&quot;erlang&quot;/&gt;&lt;project name=&quot;icu4c&quot; path=&quot;icu4c&quot; revision=&quot;26359393672c378f41f2103a8699c4357c894be7&quot; remote=&quot;couchbase&quot;/&gt;&lt;project name=&quot;snappy&quot; path=&quot;snappy&quot; revision=&quot;5681dde156e9d07adbeeab79666c9a9d7a10ec95&quot; remote=&quot;couchbase&quot;/&gt;&lt;project name=&quot;v8&quot; path=&quot;v8&quot; revision=&quot;447decb75060a106131ab4de934bcc374648e7f2&quot; remote=&quot;couchbase&quot;/&gt;&lt;project name=&quot;gperftools&quot; path=&quot;gperftools&quot; revision=&quot;8f60ba949fb8576c530ef4be148bff97106ddc59&quot; remote=&quot;couchbase&quot;/&gt;&lt;project name=&quot;pysqlite&quot; path=&quot;pysqlite&quot; revision=&quot;0ff6e32ea05037fddef1eb41a648f2a2141009ea&quot; remote=&quot;couchbase&quot;/&gt;&lt;/manifest&gt; MB-6967

dev-views index seems not to be built (at least for _count reduce) during rebalance with consistent view (1DD, 4 views) - build 1868

Bug Major Resolved Fixed Public MC Brown Sharon Barr 2.0-release-notes Thu, 18 Oct 2012 19:36:52 -0500 Thu, 6 Dec 2012 16:48:53 -0600 Wed, 21 Nov 2012 09:59:40 -0600 2.0-beta-2 2.0 documentation ns_server view-engine 0 5 More info: <br/> at the end of the rebalance, there were 630K items on this view, and it seems that the index was rebuilt until it reached the 1M number. <br/> <br/> it seems that the index is NOT being built during the rebalance. <br/> 4 logs are 70MB compressed. let me know where you want them to be uploaded. <br/> i am continuing to mess up with this cluster. 4 logs are 70MB compressed. let me know where you want them to be uploaded. <br/> i am continuing to mess up with this cluster. and another important information. there was NO load on the system. so the consistent view feature is not really have an impact here. Attached are logs from the new nodes that entered the cluster. <br/> logs from the original nodes are too large to attache. <br/> <br/> The accurate observation is that at least the reduce views are not being (or partially) built. the count drops during the rebalance and stay low after it&#39;s done. once i trigger the full view, it is being built up again. <br/> <br/> <br/> Attached are logs from the new nodes that entered the cluster. <br/> logs from the original nodes are too large to attache. <br/> <br/> The accurate observation is that at least the reduce views are not being (or partially) built. the count drops during the rebalance and stay low after it&#39;s done. once i trigger the full view, it is being built up again. <br/> <br/> <br/> What I see in the logs, is that the initial index build is being done during rebalance. <br/> <br/> During rebalance, vbucket state changes are happening all the time, which cause the updater to be stopped and restarted after each state transition. Remember that the initial index build is not resumable, so each restart means it starts from scratch. <br/> <br/> I don&#39;t think there&#39;s anything that can be done here. Only way I see, is if it were possible for ns_server to know what&#39;s the final state for all vbuckets in each node, and then do a single bulk state transition request to the indexes. <br/> <br/> This is something that was always possible since the faster initial index build method was added months ago. Couldn&#39;t access the cluster. Reproduced the same behavior on Centos64 with latest build(1870). Attaching the diags from 4 nodes. <br/> <br/> Steps followed: <br/> <br/> Load 500K items in default bucket <br/> Rebalance 1-&gt;2 nodes <br/> Create 1 dev view, <br/> map <br/> function (doc, meta) { <br/> &nbsp;&nbsp;emit(meta.id, null); <br/> } <br/> <br/> reduce <br/> _count <br/> <br/> Query full_set to create index <br/> curl -X GET &#39;<a href="http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&connection_timeout=60000&limit=10&skip=0&#39;">http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&amp;connection_timeout=60000&amp;limit=10&amp;skip=0&amp;#39;</a> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:500000} <br/> ] <br/> } <br/> <br/> <br/> Rebalance 2-&gt;4 <br/> <br/> During rebalance, query full_set returns &lt; 500K in the output <br/> <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:~# curl -X GET &#39;<a href="http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&connection_timeout=60000&limit=10&skip=0&#39;">http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&amp;connection_timeout=60000&amp;limit=10&amp;skip=0&amp;#39;</a> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:472640} <br/> ] <br/> } <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:~# curl -X GET &#39;<a href="http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&connection_timeout=60000&limit=10&skip=0&#39;">http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&amp;connection_timeout=60000&amp;limit=10&amp;skip=0&amp;#39;</a> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:453125} <br/> ] <br/> } <br/> <br/> After rebalance &amp; indexing is finished, full 500K are returned <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:~# curl -X GET &#39;<a href="http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&connection_timeout=60000&limit=10&skip=0&#39;">http://10.3.3.95:8092/default/_design/dev_d1/_view/v1?full_set=true&amp;connection_timeout=60000&amp;limit=10&amp;skip=0&amp;#39;</a> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:500000} <br/> ] <br/> } Actually, it&#39;s not doing initial build here, but what I said before it&#39;s still valid. <br/> <br/> For node .117 (node 1), what I see is that the index update is being stopped and restarted every 1 or 2 seconds due to vbucket state transition requests from ns_server: <br/> <br/> <a href="https://friendpaste.com/4qH8PqvWO4tpGANlSxp0n3">https://friendpaste.com/4qH8PqvWO4tpGANlSxp0n3</a> <br/> <br/> In each transition, only one vbucket state changes. <br/> <br/> I don&#39;t know if things could be better batched or not in ns_server. Deep, <br/> <br/> in your testing were you running load during rebalancing ? <br/> did you make sure index update was completed before rebalancing ? Deep , Iryna and I had a conversation about this scenario and we will discuss with Filipe and update the ticket to understand consistent views and index updating better I didn&#39;t run any load during rebalancing. After rebalance is finished, I can see indexing running if I do full_set queries. Every 1 or 2 seconds should be impossible, Filipe. At least for main index. We are supposed to wait for index building completion for each bucket movement. <br/> <br/> Incoming vbucket transfers are not synchronized w.r.t. each other and it&#39;s possible to have _some_ config changes due to that: i.e. we started moving _in_ vbucket 0 and 1 second later we can start moving _in_ vbucket 1, which AFAIK will cause all indexing progress to be thrown out if it&#39;s initial index building. <br/> <br/> But there&#39;s inherent limit of concurrent incoming vbucket movements. Constant changes should be impossible. <br/> <br/> Replica index is somewhat different currently. My code doesn&#39;t wait for replica indexing completion. Which I&#39;ll be happy to fix. Does couch_set_view:monitor_partition_update/3 work for replica indexes as well? Well, unless the log timestamps are not well calculated, I see at least 1 state transition per second in average: <br/> <br/> <a href="https://friendpaste.com/1Aoo4eeFhzLGFlpAW6lHQI">https://friendpaste.com/1Aoo4eeFhzLGFlpAW6lHQI</a> <br/> <br/> And no, it&#39;s not possible to wait for replica indexing of particular vbuckets like it is for main index. <br/> <br/> I see lots (if not all, or the majority) of transitions only changing one vbucket state. For example: <br/> <br/> [couchdb:info,2012-10-19T0:39:28.779,<a href='mailto:ns_1@10.176.145.104'>ns_1@10.176.145.104</a>:&lt;0.20650.1&gt;:couch_log:info:39]Set view `default`, main group `_design/dev_test`, partition states updated <br/> active partitions before: [256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510] <br/> active partitions after: [256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510] <br/> passive partitions before: [] <br/> passive partitions after: [511] <br/> cleanup partitions before: [] <br/> cleanup partitions after: [] <br/> unindexable partitions: [] <br/> replica partitions before: [] <br/> replica partitions after: [] <br/> replicas on transfer before: [] <br/> replicas on transfer after: [] <br/> pending transition before: <br/> &nbsp;&nbsp;active: [] <br/> &nbsp;&nbsp;passive: [] <br/> &nbsp;&nbsp;unindexable: [] <br/> pending transition after: <br/> &nbsp;&nbsp;active: [] <br/> &nbsp;&nbsp;passive: [] <br/> &nbsp;&nbsp;unindexable: [] <br/> <br/> <br/> Here it made one change only, added vbucket 511 to the passive state. <br/> <br/> Then 4 seconds later it does a single change, to move vbucket 511 from the passive state to the active state: <br/> <br/> [couchdb:info,2012-10-19T0:39:31.262,<a href='mailto:ns_1@10.176.145.104'>ns_1@10.176.145.104</a>:&lt;0.20650.1&gt;:couch_log:info:39]Set view `default`, main group `_design/dev_test`, partition states updated <br/> active partitions before: [256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510] <br/> active partitions after: [256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511] <br/> passive partitions before: [511] <br/> passive partitions after: [] <br/> cleanup partitions before: [] <br/> cleanup partitions after: [] <br/> unindexable partitions: [] <br/> replica partitions before: [] <br/> replica partitions after: [] <br/> replicas on transfer before: [] <br/> replicas on transfer after: [] <br/> pending transition before: <br/> &nbsp;&nbsp;active: [] <br/> &nbsp;&nbsp;passive: [] <br/> &nbsp;&nbsp;unindexable: [] <br/> pending transition after: <br/> &nbsp;&nbsp;active: [] <br/> &nbsp;&nbsp;passive: [] <br/> &nbsp;&nbsp;unindexable: [] <br/> <br/> <br/> I understand it might not be possible for ns_server to batch things better, but this doesn&#39;t help. <br/> <br/> I can revert some optimizations done several months ago for query performance, which will increase the incremental indexing checkpoint frequency. While they will likely help for such small datasets (and decreasing query performance on high concurrency scenarios), for large datasets, or when there&#39;s much load on the system (many ddocs, xdcr, bucket compaction, etc etc) we&#39;ll very likely run into the same scenario as here. <br/> <br/> When I said impossible, I was referring to initial index building. In which case waiting for single vbucket will obviously wait for all of them. <br/> <br/> For some next release we&#39;ll definitely seek better ways to interact with indexes. The problem, as I pointed our earlier, is that from our high-level perspective we&#39;re not very aware of performance implication of what we do. btw incorrect _count doesn&#39;t imply index is not being built. I recommend actually checking if results you expect are there. We have that somewhat controversial &quot;excluding&quot; of reduction values during non-steady-state. And I don&#39;t know if we heavily tested this path. the user in this case is trying a reduce query which should return the count as expected. <br/> <br/> when you say we have controversial excluding of reduction values during non-steady state ? is that expected behavior in view-engine ? <br/> <br/> do we expect users to see the reduction to return inconsistent results but the view results alway be consistent ? <br/> We expect it to be consistent. When I said &#39;somewhat controversial&#39; I was pointing out that it actually has to do quite a bit of work to return reduction value in non-steady state, compared to steady state. And I was trying to say I have no idea if we used to test this path a lot. Please take into account that the issue can be reproduced only with development views, production views are working ok. Apparently this only happens for development views, which are not supposed to be accurate across a cluster and rebalances. looked at diags posted by Sharon and confirmed they are also on dev views <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{design_documents,[&lt;&lt;&quot;_design/dev_test&quot;&gt;&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{indexer_type,main}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{set,&lt;&lt;&quot;default&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{signature,&lt;&lt;&quot;701550d970bd93f8c7c9736d27d90b51&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{started_on,1350606438}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,blocked_indexer}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{updated_on,1350606438}], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{pid,&lt;&lt;&quot;&lt;0.4324.6&gt;&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{design_documents,[&lt;&lt;&quot;_design/dev_test&quot;&gt;&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{indexer_type,main}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{set,&lt;&lt;&quot;default&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{signature,&lt;&lt;&quot;701550d970bd93f8c7c9736d27d90b51&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{started_on,1350606564}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,blocked_indexer}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{updated_on,1350606564}], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{pid,&lt;&lt;&quot;&lt;0.18271.6&gt;&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{design_documents,[&lt;&lt;&quot;_design/dev_test&quot;&gt;&gt;]}, im not quite sure why this only happens on dev views and not on production views though. if this happens for dev views we should add this to release notes that user might get incnsistent results during rebalancing with development views From the view engine perspective, there&#39;s no distinction (or concept) about development and production views. They&#39;re all treated the same way. <br/> ns_server on the other hand gives different treatment to them, like for example not triggering index updates for development views. Alk,<br/> <br/> form ns_server point of view w.r.t rebalancing and consistent views do we treat dev views any differently Farshid, I could not understand you question completely.<br/> <br/> Pure dev views are not at all affected during rebalance. After all they only cover single vbucket. Even if this vbucket is moved I believe it&#39;ll safely use other one.<br/> <br/> If we&#39;re speaking about dev views will full_set option set to true, they are no different than any production views. according to the bug description we are getting inconsistent results during rebalancing but does not happen for production views. <br/> <br/> the tester also waits until index is built before running the rebalance and full_Set is used for all view queries. <br/> if there is no difference between dev views and productions views during rebalancing then we should not be seeing different behavior. Well, there is is _no_ difference with dev with full_set and production views. Thus it appears that this bug is genuine and needs to be fixed. ok thanks. Alk, if you think this is a genuine bug, can you please review and do some triage - let me know what we should do with this bug? Ok. I was somewhat wrong. Here&#39;s full explanation.<br/> <br/> As per Frank&#39;s original idea, development design docs are supposed to be used on subset of data. People are supposed to play with map/reduce functions without throwing entire cluster&#39;s power on building full index. When that development process is done, people can try it own on full subset few times. That would build full index. We expect that once this is done, people are satisfied and will publish dev_ design document to production. Our (and stock) indexes have a nice feature where same ddocs are represented by same one index file. So work spent on building full index will not be lost as production design doc will be 100% same as just indexed in full mode dev_ design doc.<br/> <br/> Key point is, system explicitly avoids automagically building or updating that full index underneath dev_ design docs. Because that would waste cluster&#39;s resources without explicit human request for that.<br/> <br/> Thus indeed as part of rebalance we trigger and wait for index updates for all non-dev design docs. Explicitly skipping development ddocs. Even then user should still be able to see consistent result with stale=false queries, which would force index update before returning results.<br/> <br/> So indeed not a bug and perhaps worth documenting. See above Thanks Aliaksey. <br/> <br/> Based on Aliaksey&#39;s explanation, marking this non-blocker and for MC to document this non-obvious but &quot;working as designed&quot; behavior for dev-views. please note that this behavior is only expected for development views ( full_set = True and partial set ) . It does not apply for production views. Documentation has been updated with this information in both the view types, and auto-update sections of the manual. Added to RN as: <br/> <br/> &quot;If you are using development views, be aware you may see <br/> inconsistent results if you query a development view during rebalance. For production views, <br/> you are able to query during rebalance and get results consistent with those you would have recieved <br/> if no rebalance were occurring.&quot; Date of First Response Fri, 19 Oct 2012 04:59:05 -0500 Rank 3694 Time In Status