{"id":2063,"date":"2015-06-05T17:06:03","date_gmt":"2015-06-05T17:06:02","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=2063"},"modified":"2023-06-27T22:47:44","modified_gmt":"2023-06-28T05:47:44","slug":"mongodb-is-special-benchmarks-prove-it","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/mongodb-is-special-benchmarks-prove-it\/","title":{"rendered":"What does MongoDB have against transparency?"},"content":{"rendered":"

MongoDB published another benchmark performed by United Software Associates.<\/p>\n

Benchmarks are a useful tool to evaluate database performance<\/a>. But to be useful, they must be transparent and repeatable. If they fail to meet these standards, the results are questionable.<\/p>\n

In recent benchmarks, Couchbase and MongoDB took two different approaches. Couchbase clearly documented the full configuration and<\/em> included the results of every test. MongoDB did not.<\/p>\n

The below tables illustrate the different approaches taken by Avalon Consulting, LLC and United Software Associates:<\/strong><\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/td>\nUnited Software Associates<\/td>\nAvalon Consulting, LLC<\/td>\n<\/tr>\n
YCSB<\/strong><\/td>\n<\/tr>\n
Entries \/ Operations<\/td>\n400M \/ 100M<\/td>\n300M \/ 100M<\/td>\n<\/tr>\n
Value Size<\/td>\nMissing<\/strong><\/td>\n1K<\/td>\n<\/tr>\n
Data Set Size<\/td>\nMissing<\/strong><\/td>\n286GB<\/td>\n<\/tr>\n
Request Distribution<\/td>\nZipfian<\/td>\nUniform<\/td>\n<\/tr>\n
Database<\/strong><\/td>\n<\/tr>\n
Nodes<\/td>\nMissing<\/strong><\/td>\nMongoDB: 9
\nCouchbase Server: 9<\/td>\n<\/tr>\n
Replicas<\/td>\nMissing<\/strong><\/td>\n3 (1 primary, 2 secondary)<\/td>\n<\/tr>\n
Replication<\/td>\nMissing<\/strong><\/td>\nMongoDB: Async
\nCouchbase Server: Async<\/td>\n<\/tr>\n
Persistence<\/td>\nMissing<\/strong><\/td>\nMongoDB: Async
\nCouchbase Server: Async<\/td>\n<\/tr>\n
Configured Memory (per Node)<\/td>\nMissing<\/strong><\/td>\nMongoDB: 30GB
\nCouchbase Server: 30GB<\/td>\n<\/tr>\n
Total Data Set Size (with Replicas)<\/td>\nMissing<\/strong><\/td>\n858GB<\/td>\n<\/tr>\n
Primary Data Resident in Memory<\/td>\nMissing<\/strong><\/td>\n32%<\/td>\n<\/tr>\n
Version<\/strong><\/td>\n<\/tr>\n
Database<\/td>\nMongoDB: 3.0.3
\nCouchbase Server: 3.0.2<\/td>\n		MongoDB: 3.0.0
\nCouchbase Server: 3.0.2<\/td>\n<\/tr>\n
Client<\/td>\nMongoDB: 3.0.0
\nCouchbase Server: 2.1.2<\/td>\n		MongoDB: 2.1.3
\nCouchbase Server: 2.1.0<\/td>\n<\/tr>\n
Hardware<\/strong><\/td>\n<\/tr>\n
Servers<\/td>\nDatabase: 3
\nYCSB: 1<\/td>\n		Database: 9x AWS EC2 i2.2xlarge
\nYCSB: 2-23x AWS EC2 r2.8xlarge<\/td>\n<\/tr>\n
Processors<\/td>\nBoth: 2x 3.0GHz<\/td>\nDatabase: 8 vCPU (2.5GHz)
\nYCSB: 32vCPU (2.5GHz)<\/td>\n<\/tr>\n
Memory<\/td>\nBoth: 96GB<\/td>\nDatabase: 61GB
\nYCSB: 244GB<\/td>\n<\/tr>\n
Storage<\/td>\nBoth: 2x 960GB SSD<\/td>\nDatabase: 2x 800GB SSD
\nYCSB: 2x 320GB SSD<\/td>\n<\/tr>\n
Networking<\/td>\nBoth: 10GbE<\/td>\nBoth: High<\/td>\n<\/tr>\n
OS<\/td>\nBoth: Ubuntu 14.10<\/td>\nDatabase: CentOS 6
\nYCSB: Amazon Linux<\/td>\n<\/tr>\n
OS<\/strong><\/td>\n<\/tr>\n
Transparent Huge Pages (THP)<\/td>\nDisabled<\/td>\nDisabled<\/td>\n<\/tr>\n
NUMA<\/td>\nDisabled<\/td>\nDisabled<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

I hope MongoDB and DataStax will help me improve this template for future benchmarks.<\/em><\/p>\n

United Software Associates Benchmark Results<\/strong><\/h3>\n

They did not publish the results of all<\/em> tests, they published the results of one.<\/p>\n

The benchmark states the ideal number of threads for every database was 150 or 350 depending on the workload, but it does not state the number of threads for the results published.<\/p>\n

Workload A Throughput<\/u><\/h4>\n
\n\n\n\n\n\n\n\n\n\n\n\n
Threads<\/strong><\/td>\nMongoDB<\/strong><\/td>\nCouchbase Server<\/strong><\/td>\n<\/tr>\n
105<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
140<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
175<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
210<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
245<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
280<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
315<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n
350<\/td>\nMissing<\/td>\nMissing<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Results – Workload A Latency<\/u><\/h4>\n
\n\n\n\n\n\n\n\n\n\n\n\n
Threads<\/strong><\/td>\nMongoDB<\/strong><\/td>\nCouchbase Server<\/strong><\/td>\n<\/tr>\n
105<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
140<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
175<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
210<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
245<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
280<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
315<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n
350<\/td>\nRead: Missing
\nWrite: Missing<\/td>\n		Read: Missing
\nWrite: Missing<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Avalon Consulting, LLC Benchmark Results<\/h3>\n

They published the results of all<\/em> tests and<\/u> stated the number of threads for them.<\/p>\n

Workload A Throughput<\/u><\/h4>\n
\n\n\n\n\n\n\n\n\n\n\n\n
Threads<\/strong><\/td>\nMongoDB<\/strong><\/td>\nCouchbase Server<\/strong><\/td>\n<\/tr>\n
105<\/td>\n61K<\/td>\n110K<\/td>\n<\/tr>\n
140<\/td>\n65K<\/td>\n141K<\/td>\n<\/tr>\n
175<\/td>\n67K<\/td>\n154K<\/td>\n<\/tr>\n
210<\/td>\n70K<\/td>\n170K<\/td>\n<\/tr>\n
245<\/td>\n74K<\/td>\n193K<\/td>\n<\/tr>\n
280<\/td>\nExceeded Max Latency<\/td>\n238K<\/td>\n<\/tr>\n
315<\/td>\nExceeded Max Latency<\/td>\n245K<\/td>\n<\/tr>\n
350<\/td>\nExceeded Max Latency<\/td>\n252K<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Results – Workload A Latency <\/u><\/h4>\n
\n\n\n\n\n\n\n\n\n\n\n\n
Threads<\/strong><\/td>\nMongoDB<\/strong><\/td>\nCouchbase Server<\/strong><\/td>\n<\/tr>\n
105<\/td>\nRead: 1.42ms
\nWrite: 2.05ms<\/td>\n		Read: .78ms
\nWrite: .76ms<\/td>\n<\/tr>\n
140<\/td>\nRead: 2.01ms
\nWrite: 2.97ms<\/td>\n		Read: .79ms
\nWrite: .78ms<\/td>\n<\/tr>\n
175<\/td>\nRead: 3.16ms
\nWrite: 3.54ms<\/td>\n		Read: .89ms
\nWrite: .88ms<\/td>\n<\/tr>\n
210<\/td>\nRead: 3.5ms
\nWrite: 4.49ms<\/td>\n		Read: .93ms
\nWrite: .92ms<\/td>\n<\/tr>\n
245<\/td>\nRead: 4.19ms
\nWrite: 5.38ms<\/td>\n		Read: .92ms
\nWrite: .91ms<\/td>\n<\/tr>\n
280<\/td>\nExceeded Max Latency<\/td>\nRead: .92ms
\nWrite: .92ms<\/td>\n<\/tr>\n
315<\/td>\nExceeded Max Latency<\/td>\nRead: 1.06ms
\nWrite: .99ms<\/td>\n<\/tr>\n
350<\/td>\nExceeded Max Latency<\/td>\nRead: 1.22ms
\nWrite: 1.22ms<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Benchmark Configuration<\/strong><\/h2>\n

A benchmark, if you care, should answer the following questions:<\/p>\n