{"id":18116,"date":"2026-05-18T11:39:06","date_gmt":"2026-05-18T18:39:06","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=18116"},"modified":"2026-05-18T11:39:06","modified_gmt":"2026-05-18T18:39:06","slug":"can-text-to-sql-benchmarks-work-on-document-databases-a-couchbase-architecture-case-study","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/pt\/can-text-to-sql-benchmarks-work-on-document-databases-a-couchbase-architecture-case-study\/","title":{"rendered":"Can Text-to-SQL Benchmarks Work on Document Databases? A Couchbase Architecture Case Study"},"content":{"rendered":"

Executive Summary<\/span><\/h2>\n

Industry-standard text-to-SQL benchmarks are designed for relational, structured databases. However, not all real-world data workloads are confined to relational systems. Modern AI-driven query platforms increasingly operate on document-oriented data stores such as Couchbase, where schemas are flexible and data is represented as nested JSON rather than normalized tables. This divergence introduces a fundamental evaluation challenge: how can we rigorously measure the accuracy of an AI query system on a non-relational platform without rewriting the benchmark itself? Evaluating AI query systems built on non-relational platforms \u2013\u00a0 such as Couchbase \u2013 against these benchmarks therefore requires non-trivial architectural adaptation. This document presents the approach taken to re-architect the <\/span>Spider2-Lite<\/b> benchmark pipeline to run on <\/span>Couchbase<\/b>, a document-oriented, non-relational database, while fully preserving the integrity of the evaluation methodology.<\/span><\/p>\n

Why Spider2 and Spider2-Lite?<\/span><\/h2>\n

Evaluating an AI-powered natural language query system requires a benchmark that is both realistic in query complexity and widely accepted by the research community. For this work, the Spider benchmark family was selected because it is one of the most rigorous and commonly used datasets for evaluating text-to-SQL systems.<\/span><\/p>\n

Spider introduced a challenging evaluation paradigm in which models must generalize to previously unseen database schemas. Instead of memorizing query templates, systems must interpret a natural language question, understand the provided schema, and generate a correct query dynamically. This property makes Spider particularly well-suited for evaluating production systems such as Couchbase Capella iQ, where queries must operate over arbitrary customer schemas rather than a fixed training dataset.<\/span><\/p>\n

More recently, Spider2 was introduced to reflect modern data environments and higher query complexity. Spider2 expands beyond simple relational tasks and introduces queries that more closely resemble real analytical workloads. However, the full Spider2 benchmark spans multiple database backends \u2013 including BigQuery, Snowflake, and Google Analytics \u2013 which require external infrastructure and large-scale data environments.<\/span><\/p>\n

For this architectural study, the focus was placed on Spider2-Lite, a curated subset of Spider2 designed to preserve the benchmark\u2019s complexity while remaining runnable in a controlled local environment. Spider2-Lite includes a set of SQLite-backed instances that can be executed locally without cloud dependencies, making it feasible to migrate the underlying data and reproduce the evaluation pipeline.<\/span><\/p>\n

This made Spider2-Lite an ideal candidate for this case study: it maintains the rigor and schema generalization challenges of modern text-to-SQL benchmarks, while allowing the underlying relational datasets to be syst<\/span>ematically migrated to Couchbase for evaluation.<\/span><\/p>\n

The Architectural Problem<\/span><\/h2>\n

\"\"<\/p>\n

Spider2-Lite Assumptions<\/span><\/h3>\n

Spider2-Lite, like virtually all text-to-SQL benchmarks, is built on a set of foundational assumptions rooted in the relational model:<\/span><\/p>\n