The COVID-19 pandemic has substantially increased the pace of digitization, and for many organizations, data demands have never been higher. As enterprises accelerate their digital transformation journeys, the need to quickly and efficiently scale their database infrastructure to meet increased processing demands has driven the adoption of DBaaS solutions.  This blog will summarize the comparison of three of the leading DBaaS, NoSQL solutions.

Altoros is a global IT service provider that has been comparing databases for 2 decades. In its latest NoSQL performance benchmark report, Altoros has provided a comparative analysis of three NoSQL DBaaS offerings: Couchbase Cloud, MongoDB Atlas, and Amazon DynamoDB. Its aim was to measure the relative performance in terms of latency and throughput that each database can achieve, by examining performance on three different cluster configurations – 6, 9, and 18 nodes – as well as under four different workloads.

Altoros Report on NoSQL Benchmark Tests 2020

To make the tests as comprehensive as possible, the workloads varied substantially. The first was update-heavy, invoking 50 percent reads and 50 percent updates of the data. The second performed a short-range scan, involving 95 percent scans and 5 percent updates, while the third workload represented a pagination query with a single filtering option to which an offset and limit were applied. Finally, the fourth workload was a JOIN query with grouping and ordering applied.

So how did the three DBaaS providers compare?

Head-to-head on NoSQL query workloads

While each of the three DBaaS offerings had its strengths and weaknesses, across every workload test Couchbase Cloud emerged as the most robust of the three. For the first, update-heavy workload, Couchbase Cloud demonstrated low latency times across all cluster sizes. While it posted similar results as Amazon DynamoDB when measured on a 6-node cluster, Couchbase Cloud stood out on 9-node and 18-node clusters with latency times of less than a millisecond.

When measured on a short-range scan workload, Couchbase Cloud again stood out. While MongoDB Atlas performed well here, growing efficiently as more shards were added, Couchbase Cloud demonstrated the best results with a gradual increase in operations per second, as well as the lowest and most stable latency.

The results for the third workload were more balanced between MongoDB Atlas and Couchbase Cloud. Both returned similar results on a 6-node cluster and a 9-node cluster, where throughput was low since the size of the dataset and indexes doubled. Despite the similar results, Couchbase Cloud was still able to outperform MongoDB Atlas across all cluster sizes on this workload.

Meanwhile, Amazon DynamoDB was unable to compete entirely: its indexing process works in such a way that requires complex full table scans, which took too long for this workload, returning time-out errors.

In the final workload category, the results once again favored Couchbase Cloud. Couchbase Cloud posted higher throughput with 750 ops/sec on 6-node clusters, compared with 100 ops/sec for MongoDB Atlas. As with the pagination workload, Amazon DynamoDB was ineligible to compete due to the nature of its indexing process.

Here is one of the workloads compared in the report – for the three cluster sizes, measuring throughput and latency:

Altoros NoSQL Benchmarking Report for Couchbase DynamoDB MongoDB

Figure 4.1.3 Performance results under Workload A on 6-, 9-, and 18-node clusters

Amazon DynamoDB had unstable results because it produced a great number of failed operations. On each type of cluster, Amazon DynamoDB had 40% of failed update operations and almost 1% of failed read operations.

The results are in – best performing document store

Across the board, Altoros’ performance benchmark found Couchbase Cloud to be the best performing document store DBaaS available. MongoDB Atlas produced decent results, but fell short on JOIN operations for sharded collections out of the box:

NoSQL benchmark for JOIN operations

Amazon DynamoDB also performed well on certain workloads but throws exceptions when read/write operations exceed the predetermined capacities which resulted in a large volume of failed operations in certain workloads. Comparatively, the query engine of Couchbase Cloud supports aggregation, filtering, and JOIN operations without the need to model data for each specific query, ensuring a good level of scalability across all of these operations. 

So, it’s Couchbase with the win!


Posted by Rick Jacobs

Rick Jacobs is the Technical Product Marketing Manager at Couchbase. His varied background includes experience at many of the world’s leading organizations such as Computer Sciences Corporation, IBM, Cloudera etc. He comes with over 15 years of general technology experience garnered from serving in development, consulting, data science, sales engineering and technical marketing roles. He holds several academic degrees including an MS in Computational Science from George Mason University.

Leave a reply