{"id":2249,"date":"2016-05-19T22:59:51","date_gmt":"2016-05-19T22:59:51","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=2249"},"modified":"2025-10-09T07:13:03","modified_gmt":"2025-10-09T14:13:03","slug":"faster-indexing-and-query-with-memory-optimized-global-secondary-indexes-gsi-part-ii","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/faster-indexing-and-query-with-memory-optimized-global-secondary-indexes-gsi-part-ii\/","title":{"rendered":"Faster Indexing and Query: Introducing Memory Optimized Global Secondary Indexes – PART II"},"content":{"rendered":"

In PART I<\/a>, we have covered the differences in indexing with local and global indexes. We have also concluded with the 2 storage options Couchbase Server provides for global secondary indexes: standard GSI and memory\u00a0optimized GSI. Lets dive into memory optimized GSI now;<\/p>\n

What is a Memory Optimized Global Secondary Index?<\/h2>\n

In previous post covering global vs local indexes<\/a>, we talked extensively about how global indexes reduce query latency vs local indexes. We also saw how challenging it can be to maintain these global indexes: Global indexes\u00a0require a subset of nodes to keep up with a large cluster of nodes. In the case of Couchbase Server, 10-100K ops\/sec is the norm for a cluster. <\/span><\/p>\n

Challenges\u00a0do\u00a0not end there!\u00a0Global indexing can be even more challenging in cases like array-indexes (read more about array indexes\u00a0<\/span>here<\/a>). Array indexes index the elements of an <\/span>embedded\u00a0array in a JSON document and the mutations to documents with arrays\u00a0amplify a document<\/span>\u00a0to many index writes.\u00a0<\/span>Memory optimized global secondary index (MOI) is purpose built to solve the challenges\u00a0for these most demanding applications –\u00a0travel-itinerary, score-board\u00a0<\/em>and\u00a0fraud detection<\/em>\u00a0– no problem!<\/p>\n

MOI can provide over 10x better latency and throughput under faster mutations to data compared to standard GSI\u00a0and here is how it does that:<\/p>\n

    \n
  1. Lock-free processing for indexing\u00a0simply allows massive concurrency when maintaining\u00a0the incoming mutation to the index.<\/li>\n
  2. Skiplist structure optimize in-memory storage, as opposed to B+Tree indexes.<\/li>\n
  3. Forcing the index into memory storage, means MOI does not run at disk speeds for storing the index – instead it take regular snapshots to disk for recovery only.<\/li>\n<\/ol>\n

    Memory Optimized vs. Standard Global Secondary Indexes<\/h3>\n

    Memory optimized indexes are added in 4.5 as an additional storage option for GSIs. Standard global secondary indexes have been there since version 4.0.\u00a0Administrators can configure GSI with either the standard GSI storage, which uses ForestDB underneath, for indexes that cannot fit in memory\u00a0or can pick the memory optimized GSI for faster in-memory\u00a0indexing and queries.<\/p>\n

    Typically, indexes are created to lower query latencies and keeping indexes in memory reduces latencies a great deal more! MOI is designed for lower latency and <\/span>highest\u00a0throughput needs and\u00a0MOI require machines with large memory to keep the index in RAM. Standard GSI can spill to disk when memory runs out. IO Subsystem performance becomes <\/span>extremely\u00a0important for standard GSI to be able to perform\u00a0well.\u00a0However unlike standard GSI, h<\/span>igh performance IO subsystem is not required for MOI.\u00a0As MOI runs at in-memory speeds, initial and ongoing indexing times are faster with MOI compared to Standard GSI.\u00a0<\/span><\/p>\n

    The following table summarizes the major differences between standard vs memory optimized GSIs;<\/p>\n

    <\/p>\n

    Creating and Managing Memory Optimized GSIs<\/h3>\n

    Regardless of the store type, CREATE INDEX is the way to create global secondary indexes in Couchbase Server. In fact there are no MOI specific options in CREATE INDEX statement.\u00a0In general, high-availability and partitioning mechanics stay the same with standards vs memory optimized GSI. However it is important to note that MOI comes with additional stats and alerts to help with placement of indexes and with managing indexes.<\/p>\n

    Placing Memory Optimized Indexes in Couchbase Server Cluster<\/h4>\n

    Memory optimized indexes provide 2 important stats that can guide placement of MOI with the NODES clause on CREATE INDEX<\/a>.<\/p>\n