[MB-7522] Optimize very common index state transitions (rebalance) Created: 11/Jan/13  Updated: 17/Jan/13  Resolved: 11/Jan/13

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.0.1
Fix Version/s: 2.0.1
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Filipe Manana Assignee: Filipe Manana
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Optimize 2 types of state transitions:
    
    1) If a state transition consists only of moving partitions
    between the passive and active states (or vice-versa), that
    is, no new partitions are added nor existing partitions are
    marked for cleanup, don't restart the updater (at the expense
    of more complex logic to correct snapshots received from a
    currently running updater).
    
    2) If a state transition only adds new partitions to the
    passive state (as ns_server's rebalance frequently calls),
    don't restart the updater, and send the updater the new
    list of passive partitions, so that it has a chance to
    iterate over those partitions' content and index them in
    the same run.
    
    As long as the rebalancer in ns_server doesn't keep changing
    its logic regarding state transitions all the time, this
    provides a significant reduction of the indexing time. However
    for at least some scenarios, there's still significant portion
    of rebalance time not spent on indexing nor index compaction.

 Comments   
Comment by Thuan Nguyen [ 11/Jan/13 ]
Integrated in github-couchdb-preview #548 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/548/])
    MB-7522 Optimize index state transitions (rebalance) (Revision ac82c60302422747e8804d566211a20684ec78fb)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/test/17-unindexable-partitions.t
* src/couch_set_view/Makefile.am
* src/couch_set_view/src/couch_set_view_group.erl
* src/couch_set_view/test/21-updater-cleanup.t
* src/couch_set_view/src/couch_set_view_updater.erl
* src/couch_set_view/src/couch_set_view_util.erl
* src/couch_set_view/test/19-compaction-retry.t
* src/couch_set_view/test/05-replicas-transfer.t
* src/couch_set_view/test/07-replica-compaction.t
* src/couch_set_view/test/24-updater-add-more-passive-partitions.t
Comment by Filipe Manana [ 11/Jan/13 ]
Gerrit: http://review.couchbase.org/#/c/23871/

Some results: https://docs.google.com/document/d/1aHHhY-ami84-aEQyx42h95rlp7Vi6WTwlozaIstFgfA/edit , related to MB-6726.

Drops all needs to restart updater for the current type of transitions the rebalancer does to indexes.
Comment by Filipe Manana [ 11/Jan/13 ]
Merged.
Comment by Karen Zeller [ 11/Jan/13 ]
I'm elevating this to 2.0.1 release notes....
Comment by Thuan Nguyen [ 12/Jan/13 ]
Integrated in github-couchdb-preview #549 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/549/])
    MB-7522 Improve test 24-updater-add-more-passive-partitions.t (Revision 9126caff6319243a9b0fdfa50e05477c94092d00)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/test/24-updater-add-more-passive-partitions.t
Comment by Thuan Nguyen [ 12/Jan/13 ]
Integrated in github-couchdb-preview #550 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/550/])
    MB-7522 Fix race condition in test (Revision 84d25c7cb136c9f66adbb572c99cca81235ef13e)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/test/24-updater-add-more-passive-partitions.t
Comment by Thuan Nguyen [ 14/Jan/13 ]
Integrated in github-couchdb-preview #551 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/551/])
    MB-7522 Further test improvements/coverage (Revision 8deb85624bacdfa290adb4973f7ba1d204885309)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/test/24-updater-add-more-passive-partitions.t
Comment by Thuan Nguyen [ 17/Jan/13 ]
Integrated in github-couchdb-preview #552 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/552/])
    MB-7522 Do updater restarts if a pending transition exists (Revision 060f7ae833dd871af56d188490a6371f35509782)

     Result = SUCCESS
Filipe David Borba Manana :
Files :
* src/couch_set_view/src/couch_set_view_group.erl
Generated at Wed May 22 00:04:07 CDT 2013 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.