{"id":15552,"date":"2024-04-02T13:09:34","date_gmt":"2024-04-02T20:09:34","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=15552"},"modified":"2024-04-18T08:53:41","modified_gmt":"2024-04-18T15:53:41","slug":"file-transfer-index-rebalance","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/file-transfer-index-rebalance\/","title":{"rendered":"Rebalance Reimagined: Faster Scaling of Couchbase’s Index Service With File Transfers"},"content":{"rendered":"

Faster scaling of database resources is essential for maintaining efficient and performant databases, especially with the increased pressure of data ingestion, growing query demands, and the need to handle failovers seamlessly. As application-driven query traffic is primarily handled by index services, faster scaling of index rebalance services is critical to highly performant applications.<\/span><\/p>\n

For the index service, the scaling operation (also referred to as <\/span>rebalance<\/span><\/a>) involves moving individual indexes\/replicas\/partitions among the available index service nodes in the cluster. The aim is to minimize load imbalances and optimize resource utilization metrics, like CPU and memory, across all nodes.<\/span><\/p>\n

This article explores the limitations and improvements made to the index rebalance process in Couchbase-Server Version 7.6. It introduces a new rebalance flow based on efficient file transfers, offering significant benefits such as substantial reductions in rebalance time and optimized resource utilization, including lower CPU and memory consumption.<\/span><\/p>\n

 <\/p>\n

Overview of index service rebalance<\/span><\/h2>\n

At a high level, index service rebalance operates in 3 phases:<\/span><\/p>\n

Planning<\/span><\/h3>\n
    \n
  1. \n
      \n
    1. In the planning phase<\/a>, the information of all indexes across all nodes in the cluster is gathered, along with their load statistics. Load of an index is derived from various factors like CPU utilization, memory utilization, scan rate, mutation processing rate, disk utilization, etc.<\/span><\/li>\n
    2. An optimization algorithm is used to minimize load variance across the cluster. For example, if an index service node experiences significantly higher mutation and scan traffic compared to others, the algorithm redistributes indexes to balance the overall load and reduce variance.<\/span><\/li>\n
    3. The optimization algorithm decides the index movements from their existing nodes to new nodes in a simulated environment to arrive at a distribution that minimizes the load variance in the cluster.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      Execution phase<\/span><\/h3>\n
        \n
      1. \n
          \n
        1. Based on the final plan decided in the planning phase, indexes are moved from their existing nodes to new targets.<\/span><\/li>\n
        2. This phase is the most significant contributor to index rebalance time, directly impacted by the movement method and the number of indexes involved.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          Dropping indexes on source nodes<\/span><\/h3>\n
            \n
          1. \n
              \n
            1. Once the indexes on the new target nodes are fully rebuilt and ready to handle queries, all incoming scan requests for those indexes will be redirected to the new nodes.<\/span><\/li>\n
            2. The indexes on the existing source nodes will be removed once the corresponding indexes on the new nodes are ready to serve scan traffic.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

              The Execution phase<\/em>, currently relies on rebuilding the indexes that are being moved by re-streaming and re-processing all the documents from the data service. While this scheme works well functionally, it has these drawbacks:<\/span><\/p>\n