{"id":5777,"date":"2018-09-10T04:27:56","date_gmt":"2018-09-10T11:27:56","guid":{"rendered":"http:\/\/www.couchbase.com\/blog\/?p=5777"},"modified":"2025-06-13T20:19:57","modified_gmt":"2025-06-14T03:19:57","slug":"ycsb-json-benchmarking-json-databases-by-extending-ycsb","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/ycsb-json-benchmarking-json-databases-by-extending-ycsb\/","title":{"rendered":"Using YCSB to Benchmark JSON Databases"},"content":{"rendered":"

Bruce\u00a0Lindsay<\/a>\u00a0once said<\/a>, “There are three things important in the database world: Performance, Performance, and Performance”.\u00a0 Most enterprise architects know, as we progress in database features and architectures, it’s important to measure performance in an open way so they can compare total cost of ownership reliably.<\/p>\n

YCSB<\/a> did a great job of benchmarking datastores serving the “Cloud OLTP” applications. These data stores were simple with simple get, put, delete operations.\u00a0 The original YCSB benchmark<\/a> consists of a simple insert, update, delete, and scan operations on a simple document of 10 key-values; workloads are defined with a mix of these operations with various percentages.<\/p>\n

JSON<\/a> databases like Couchbase<\/a> and MongoDB<\/a> have a more advanced data model with scalars, nested objects, arrays, arrays of objects, arrays and arrays of objects.\u00a0 JSON databases also have more sophisticated query<\/a> language, indexes, and capabilities. In addition to CRUD operations, applications routinely use the declarative query languages in these databases to search, paginate, and run reports.\u00a0 So, to help architects to evaluate platforms effectively, we need an additional benchmark to measure these capabilities in addition to the basic CRUD operations. This YCSB tutorial explains its capabilities in filling the gap.<\/span><\/p>\n

YCSB paper<\/a> states: We also hope to foster the development of additional cloud benchmark suites that represent other classes of applications by making our benchmark tool available via open source. In this regard, a key feature of the YCSB framework\/tool is that it is extensible\u2014it supports easy definition of new workloads, in addition to making it easy to benchmark new systems.<\/p><\/blockquote>\n

This benchmark extends YCSB to JSON databases by extending existing operations to JSON and then defining new operations and new workloads.<\/p>\n

Here’s the outline.<\/span><\/strong><\/p>\n

    \n
  1. Introduction<\/span><\/li>\n
  2. Data Model<\/span><\/li>\n
  3. Benchmark Operations<\/span><\/li>\n
  4. Benchmark Workloads<\/span><\/li>\n
  5. YCSB-JSON implementation<\/span><\/li>\n
  6. How to run YCSB-JSON?<\/span><\/li>\n
  7. References<\/span><\/li>\n<\/ol>\n
    1. Introduction<\/span><\/strong><\/h5>\n

    YCSB was developed to measure the performance of scalable NoSQL key-value datastores. YCSB infrastructure does that job well.\u00a0 YCSB uses a simple flat key-value. Couchbase uses a JSON model, which customers use to massively interactive applications.\u00a0 We\u2019ve built and are building features into the product to enable customers to build these applications effectively. We need performance measurements for these use cases.<\/span><\/p>\n

    There are additional databases supporting JSON model: MongoDB, DocumentDB, DynamoDB, RethinkDB, Oracle NoSQL.\u00a0 When running YCSB on JSON databases (Couchbase, MongoDB, etc), the driver simply stores and retrieves strings in the JSON key-value structure. All of these databases require a new benchmark to measure processing of rich structure of JSON (nested objects, arrays) and operations like paging, grouping, aggregations.<\/span><\/p>\n

    The purpose of YCSB-JSON is to extend the YCSB benchmark to measure JSON database capability to cover these two things: <\/span><\/p>\n

      \n
    1. Operations representative of massively interactive applications.<\/span>\n