{"id":13804,"date":"2022-10-17T09:24:10","date_gmt":"2022-10-17T16:24:10","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=13804"},"modified":"2024-05-03T18:28:19","modified_gmt":"2024-05-04T01:28:19","slug":"magma-next-gen-document-storage-engine","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/magma-next-gen-document-storage-engine\/","title":{"rendered":"Magma: The Next-Generation Document Storage Engine"},"content":{"rendered":"

The Couchbase database platform supports two storage mechanisms: Couchstore, the default, and Magma, the recently released engine. Both offer benefits under various scenarios. This blog post gives a brief overview of the new Magma storage engine, <\/span>provides a comparison of each engine, and summarizes the results of the performance benchmarks<\/a>.<\/span>\u00a0<\/span><\/p>\n

Couchstore is a mature storage engine that is optimized for high performance with large datasets, particularly ones that can fit in memory. The minimum bucket size for Couchstore is 100MB. It\u2019s ideal for caching use cases and situations where data compression is not a primary deciding factor.\u00a0<\/span><\/p>\n

Magma is a new storage engine that is designed to be highly performant even with very large datasets that do not fit in memory. It is transcendent for use cases where disk access is paramount. Magma is optimized to run on very low amounts of memory even with very large datasets. Magma really shines when used for <\/span>datasets that will not fit into available memory and that require maximum data compression<\/span>.<\/span><\/p>\n

Below is a comparison table that summarizes each storage mechanism.<\/span><\/p>\n

Couchstore and Magma Comparison<\/span><\/h2>\n\n\n\n\n\n\n\n\n\n\n\n\n
Criteria<\/b><\/td>\nCouchstore<\/b><\/td>\nMagma<\/b><\/td>\n<\/tr>\n
Minimum bucket memory quota<\/span><\/td>\n100MB<\/span><\/td>\n1GB<\/span><\/td>\n<\/tr>\n
Minimum memory to data ratio<\/span><\/td>\n10%<\/span><\/td>\n1%<\/span><\/td>\n<\/tr>\n
Maximum data per node<\/span><\/td>\n3TB<\/span><\/td>\n10TB<\/span><\/td>\n<\/tr>\n
Data size optimization<\/span><\/td>\nBest when the working dataset can fit into memory<\/span><\/td>\nBest when the <\/span>working set is much larger than the available memory and you need disk access speed only<\/span><\/td>\n<\/tr>\n
Store and access<\/span><\/td>\nAccess data up to ~ 1TB<\/span><\/td>\nStore and access several terabytes of data<\/span><\/td>\n<\/tr>\n
Hardware<\/span><\/td>\nCan run on low-end hardware<\/span><\/td>\nQuality hardware is preferred<\/span><\/td>\n<\/tr>\n
Supported services<\/span><\/td>\nAll services including full-text search, eventing, and analytics are available<\/span><\/td>\nAll services including full-text search, eventing, and analytics are available with the 7.1.2 GA release<\/td>\n<\/tr>\n
Data persistence<\/span><\/td>\nMost data is accessed from the memory cache<\/span><\/td>\nApplications need large amounts of persistent, durable data<\/span><\/td>\n<\/tr>\n
Use cases<\/span><\/td>\nUse case primarily requires memory access<\/span><\/td>\nUse case primarily requires disk access<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Magma Overview<\/span><\/h2>\n

Magma is the next-generation document storage engine of Couchbase Server. It was designed with the goals of improving both data density and write performance on each cluster node. It achieves these goals by separating index and document data to minimize write amplification (WA). Write amplification relates to writing data to file storage where the write is multiplied due to factors like immutable data. Magma also includes an incremental compaction method to maintain space and allow for high data density and lower memory requirements. Lowering write amplification increases write throughput and also extends the life expectancy of SSDs by reducing the number of write-erase cycles.<\/span><\/p>\n

\u00a0<\/span>Other design goals for the Magma project included:<\/span><\/p>\n

Scalable concurrent compactions <\/b>\u2013 Full database compaction is an expensive operation. Managing a high-density database requires small, concurrent, incremental compactions to reclaim space.<\/span><\/p>\n

Solid state drive (SSD) optimization<\/b> \u2013 Random I\/O was minimized to only occur during point lookup operations. While sequential read and write I\/O access patterns are leveraged to take advantage of the full bandwidth of the SSDs.<\/span><\/p>\n

Low memory footprint<\/b> \u2013 High-density data decreases the possibility for read and write caching. So Magma is optimized to utilize a small memory footprint.<\/span><\/p>\n

Garbage collection<\/b> \u2013 Magma includes a method of estimating the fragmentation in the log-structured object store and accurately calculating the disk fragmentation. This is used to calculate the garbage size per log segment which triggers compactions when the fragmentation reaches a threshold of 50%.<\/span><\/p>\n

One key aspect of the Magma architecture is the log-structured object store which stores documents on an append-only segmented log. The log-structured store maintains an index that allows the querying of a document by seqno (sequence number). The object store contains log segment files that are arranged sequentially using a growing log with a tail for accepting incoming writes (see the Magma object store architecture diagram below). The background thread appends the document mutations to the tail log with each document receiving a unique segno.\u00a0<\/span><\/p>\n

\"Magma

Magma object store architecture<\/em><\/p><\/div>\n

 <\/p>\n

Although the log can contain multiple immutable document versions with the same key, the older document versions become stale when new versions are appended. The read operations always read the latest version, so when a key lookup is performed the latest version of the document is returned. Eventually, the stale records are removed from storage and a separate garbage collection process is used to reclaim space.<\/span><\/p>\n

Performance Summary<\/span><\/h2>\n

Magma was performance tested against RocksDB and Couchstore. The evaluation focused on throughput and write and space amplification for various Yahoo! Cloud Serving Benchmark (YCSB) workloads with data that was too large to fit in memory. During two rounds of testing the conclusions were:<\/span><\/p>\n

Round 1<\/span><\/p>\n