MongoDB published another benchmark performed by United Software Associates.
Benchmarks are a useful tool to evaluate database performance. But to be useful, they must be transparent and repeatable. If they fail to meet these standards, the results are questionable.
In recent benchmarks, Couchbase and MongoDB took two different approaches. Couchbase clearly documented the full configuration and included the results of every test. MongoDB did not.
The below tables illustrate the different approaches taken by Avalon Consulting, LLC and United Software Associates:
United Software Associates | Avalon Consulting, LLC | |
YCSB | ||
Entries / Operations | 400M / 100M | 300M / 100M |
Value Size | Missing | 1K |
Data Set Size | Missing | 286GB |
Request Distribution | Zipfian | Uniform |
Database | ||
Nodes | Missing | MongoDB: 9 Couchbase Server: 9 |
Replicas | Missing | 3 (1 primary, 2 secondary) |
Replication | Missing | MongoDB: Async Couchbase Server: Async |
Persistence | Missing | MongoDB: Async Couchbase Server: Async |
Configured Memory (per Node) | Missing | MongoDB: 30GB Couchbase Server: 30GB |
Total Data Set Size (with Replicas) | Missing | 858GB |
Primary Data Resident in Memory | Missing | 32% |
Version | ||
Database | MongoDB: 3.0.3 Couchbase Server: 3.0.2 |
MongoDB: 3.0.0 Couchbase Server: 3.0.2 |
Client | MongoDB: 3.0.0 Couchbase Server: 2.1.2 |
MongoDB: 2.1.3 Couchbase Server: 2.1.0 |
Hardware | ||
Servers | Database: 3 YCSB: 1 |
Database: 9x AWS EC2 i2.2xlarge YCSB: 2-23x AWS EC2 r2.8xlarge |
Processors | Both: 2x 3.0GHz | Database: 8 vCPU (2.5GHz) YCSB: 32vCPU (2.5GHz) |
Memory | Both: 96GB | Database: 61GB YCSB: 244GB |
Storage | Both: 2x 960GB SSD | Database: 2x 800GB SSD YCSB: 2x 320GB SSD |
Networking | Both: 10GbE | Both: High |
OS | Both: Ubuntu 14.10 | Database: CentOS 6 YCSB: Amazon Linux |
OS | ||
Transparent Huge Pages (THP) | Disabled | Disabled |
NUMA | Disabled | Disabled |
I hope MongoDB and DataStax will help me improve this template for future benchmarks.
United Software Associates Benchmark Results
They did not publish the results of all tests, they published the results of one.
The benchmark states the ideal number of threads for every database was 150 or 350 depending on the workload, but it does not state the number of threads for the results published.
Workload A Throughput
Threads | MongoDB | Couchbase Server |
105 | Missing | Missing |
140 | Missing | Missing |
175 | Missing | Missing |
210 | Missing | Missing |
245 | Missing | Missing |
280 | Missing | Missing |
315 | Missing | Missing |
350 | Missing | Missing |
Results – Workload A Latency
Threads | MongoDB | Couchbase Server |
105 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
140 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
175 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
210 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
245 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
280 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
315 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
350 | Read: Missing Write: Missing |
Read: Missing Write: Missing |
Avalon Consulting, LLC Benchmark Results
They published the results of all tests and stated the number of threads for them.
Workload A Throughput
Threads | MongoDB | Couchbase Server |
105 | 61K | 110K |
140 | 65K | 141K |
175 | 67K | 154K |
210 | 70K | 170K |
245 | 74K | 193K |
280 | Exceeded Max Latency | 238K |
315 | Exceeded Max Latency | 245K |
350 | Exceeded Max Latency | 252K |
Results – Workload A Latency
Threads | MongoDB | Couchbase Server |
105 | Read: 1.42ms Write: 2.05ms |
Read: .78ms Write: .76ms |
140 | Read: 2.01ms Write: 2.97ms |
Read: .79ms Write: .78ms |
175 | Read: 3.16ms Write: 3.54ms |
Read: .89ms Write: .88ms |
210 | Read: 3.5ms Write: 4.49ms |
Read: .93ms Write: .92ms |
245 | Read: 4.19ms Write: 5.38ms |
Read: .92ms Write: .91ms |
280 | Exceeded Max Latency | Read: .92ms Write: .92ms |
315 | Exceeded Max Latency | Read: 1.06ms Write: .99ms |
350 | Exceeded Max Latency | Read: 1.22ms Write: 1.22ms |
Benchmark Configuration
A benchmark, if you care, should answer the following questions:
- What was the hardware configuration?
- How was the operating system configured?
- What were the database and client versions?
- How were the databases configured?
- How was YCSB configured?
So, what configuration is this United Software Associates benchmark missing?
- The number of nodes
- The replica configuration (# of replicas
- The replication configuration (async or sync)
- The persistence configuration (async or sync)
- The size of the values
- The size of the data set
- The size of the cache (per node)
- The size of the total data set (including replicas)
- The percentage of primary data resident in memory
- The results of all runs
- The number of threads for all published runs
In addition, the GitHub repository referenced in this United Software Associates white paper does not include the client configuration for MongoDB, Cassandra, or Couchbase Server.
And what about the MongoDB routers and config servers?
Were they deployed? If so, where?
I don’t know because this benchmark does not include all of the configuration. The first benchmark included most of the configuration, but there were a number of mistakes. With this benchmark, they may have fixed the mistakes, but they failed to include all of the configuration.
Bottom Line
It’s important for benchmarks to be credible. Anyone should be able to reproduce a benchmark and validate its results. All of the configuration must be provided and, preferably, it should be performed on cloud infrastructure. If not, vendors can manipulate the configuration to make unfair comparisons look fair.
That’s not helpful to anyone.
Resources
MongoDB + United Software Associates Benchmark
White Paper | Code
Couchbase + Avalon Consulting, LLC Benchmark
White Paper | Code
Discuss on Hacker News