Why Vector-Only RAG Fails on Analytical Questions, and a Reference Architecture That Doesn’t

PRISM: a Couchbase reference architecture for governed document intelligence

The question that breaks RAG

Ask a retrieval-augmented generation (RAG) system a lookup question about a document and it usually does well. Analytical questions are where the cracks show first. Ask one over a stack of 200- to 500-page SEC filings and things fall apart. Consider a simple-sounding question:

“What was 3M’s quick ratio at the end of the third quarter of 2022?”

(The quick ratio is a liquidity measure of how well current assets cover short-term obligations.)

The textbook pipeline is familiar: chunk the PDFs, create embeddings, build a vector index, pull the top 10 chunks, and let an LLM answer. For this question, that pipeline fails three separate ways.

• Scope failure. Which company? Which filing? Which period? Unless the system resolves the right 10-Q before it searches, the top 10 chunks can come from the wrong filing, quarter, or company. Similarity search will happily return a convincing answer from a document you never asked about.
• Evidence failure. The phrase “quick ratio” does not appear in the filing. A filing prints line items such as total current assets, inventories, and total current liabilities. The question has to be translated into the printed terms before retrieval can find them, and a purely semantic search is poor at matching exact printed labels.
• Interpretation failure. Even with the right numbers in hand, there is more than one defensible formula for a quick ratio. Some conventions subtract inventory from current assets, others build the numerator from cash, receivables and securities. Depending on the convention, the same balance sheet can land on either side of a “healthy” threshold. Somebody with authority has to decide which definition counts.

Vector-only RAG is not enough for analytical questions. Answering them reliably takes four things: scope resolution, hybrid search, deterministic calculation, and governed learning. PRISM is a reference architecture that puts those four things together on Couchbase.

They are not the only areas the architecture helps. Resolving the right document, matching the labels a filing actually prints, and tying every figure to its row, column, and page make a plain lookup answer more trustworthy too. The extra machinery, including approved formulas, deterministic calculation, and review of ambiguity, only comes into play when a question needs a number computed or a judgment made.

The core idea: divide the work by who is good at what

The most important design decision in PRISM is about responsibility, not technology. Each participant does only what it can be trusted to do

• The LLM plans the scope and the evidence, proposes candidate interpretations, binds figures to the places they were found, and writes the final prose. It never performs governed arithmetic.
• The platform (Couchbase) holds the documents, vectors, catalog, vocabulary, and approved definitions in one place, and fuses lexical and vector retrieval in a single query.
• Code and humans close the loop. Code executes the arithmetic and validates every number deterministically. Humans approve definitions and curate the vocabulary.

The consequence is simple: language models are excellent at reading and writing, but are unreliable calculators. So PRISM lets the model read and write, and makes sure the numbers can be verified.

Five artifacts

PRISM organizes everything it knows into five artifacts, in two groups.

Data artifacts describe the corpus:

• Catalog: Which document? One record per source document: company, form type, fiscal period, and source filename.
• Docs: What does it say? The chunks themselves, with tables, embeddings, and page-level provenance.

Knowledge artifacts describe the domain:

• Skills: How are these documents built? Industry-wide domain conventions, such as how filings are structured. They ship with the system and are written by domain experts, with no code involved.
• Concepts: What do people call it? Your users’ vocabulary mapped to the terms printed in the documents. They are specific to a corpus.
• Dictionary: What does it mean? Approved formulas and thresholds. Humans approve every entry, and the dictionary starts empty.

The fact that the dictionary starts empty matters. PRISM does not need to be taught the domain before it is useful. When asked about a concept nobody has defined, it can still bind figures and compute candidate interpretations, labeled as candidates. When those candidates disagree in a way that changes the conclusion, PRISM declines to give a verdict and surfaces the ambiguity instead. A human then approves one convention, making the answer deterministic. The system learns through governed corrections instead of silently guessing.

Why a catalog? Why not just put the metadata on every chunk?

Good question. Each chunk could simply carry its company, form type, and period. Here are some reasons to keep a separate record per document:

1. Resolving the document is a lookup, not a similarity search. Finding “3M, 10-Q, Q3 2022” means having to choose among a few hundred small records. A catalog answers that exactly and cheaply. Once the document is known, retrieval becomes a precise filter on one filename instead of a hope that the right chunks float to the top.
2. The facts are not on most pages. A filing prints its company, form, and period on the cover page. A middle chunk from a balance sheet has none of that. Stamping it onto every chunk means repeating the same facts thousands of times, and each copy has the potential to drift or be extracted inconsistently.
3. One source of truth. If a period or filename needs to be corrected, you simply edit one catalog record. With metadata on chunks, you would have to rewrite thousands of chunks and refresh the index entries behind them.
4. Ambiguity becomes visible. With a catalog, the planner can see which filings exist. If a question could mean two filings, the system can ask or say so, instead of quietly retrieving from the wrong one.
5. Chunks stay lean. Chunks carry only what is true at the chunk level: page, row and column, and document ID. A smaller payload on every chunk keeps the vector index leaner and the documents simpler.

In short, the catalog answers “which document?” before any chunks are retrieved. The chunks then only need to answer “what does it say?”

From PDF to searchable evidence

PRISM starts with the filings as PDFs stored in Amazon S3. It deliberately does not parse or chunk them itself. That work belongs to a Couchbase AI Data Plane™ workflow, which reads the PDFs from S3, parses text, tables, and page metadata, splits them into chunks, and generates dense vector embeddings with a deployed embedding model. It also keeps provenance, including page numbers, row and column coordinates and document IDs. The results land in Couchbase Capella. At query time, the same embedding model embeds user queries, so documents and questions live in the same vector space.

A separate, lightweight pass reads each document’s first pages to build its catalog record: company, form, fiscal period, and filename. It needs no layout analysis, so it is fast. Finally, a hybrid search index is created over the chunks, combining BM25 lexical search with kNN vector search, with structured metadata filters alongside.

Hybrid retrieval in one query

This is where Couchbase does the most work. Analytical questions need two different kinds of search at the same time:

• The lexical channel (BM25 plus anchors) catches precise printed terms, identifiers, and financial labels. “Total current liabilities” must match “Total current liabilities”, not a nearby paraphrase.
• The semantic channel (kNN vector) catches paraphrases, related language, and conceptual intent, such as a user’s own words for a metric.

Couchbase 8.5 (coming soon) runs both channels in one SEARCH() call and fuses their results on the server, so the application does not have to stitch two result lists together. The fused list becomes the final top 10 evidence chunks that reach the LLM.

How the two lists are combined is a query-time choice, and there are several strategies:

Strategy	Formula / process	Key benefit	Limitation / implementation
Additive Fusion	Weighted lexical score plus weighted vector score	Simple and highly backward-compatible	Raw score scales are often not comparable
Native RRF (Reciprocal Rank Fusion)	Sum of weight / (rank constant + rank)	Ignores scale mismatches and rewards agreement between channels	Server-side, in a single Couchbase SEARCH()
Native RSF (Relative Score Fusion)	Min-max normalize each channel, then combine weighted scores	Preserves relative local score magnitude	Sensitive to outliers; native SEARCH() execution
Application RRF	Retrieve the lists separately and merge ranks in PRISM	Exposes all sub-chan	Adds application logic and an extra network trip

Two knobs shape the result. Channel weights (the top-level query boost and the kNN boost) control how much lexical versus semantic evidence matters. The candidate window controls how many results from each channel take part in fusion.

The takeaway is that fusion is a query-time decision. It is not another index to build or another ingestion pipeline to run. You can change strategy, weights, and window without re-ingesting a single document.

The governed flow, end to end

Here is how the pieces work together when a user asks the quick-ratio question.

Resolve and plan. The user’s prompt goes to the LLM together with the skills. The model works out which document type, year, and period are needed, and what to search for: subject words, the printed line items, and the candidate formulas. The catalog then turns that information into an exact filename. That is scope resolved.

Retrieve. The application enriches the search keywords using concepts, so the user’s vocabulary maps to the terms the filing prints. Couchbase then runs the hybrid query, scoped to the resolved document, and returns the top chunks.

Bind. The LLM reads the retrieved chunks and binds each figure to the specific row, column, and page it came from, with its entity, period, and units. This is the evidence trail, and every binding can be checked.

Validate. Now the dictionary comes in. If a human has approved a formula for this concept, that formula is the one that runs, and the LLM is not allowed to substitute its own. Code evaluates it deterministically over the validated bindings. If no definition has been approved, the system computes labeled candidate interpretations in code and checks whether they agree on the conclusion. When they straddle a threshold, the system declines to issue a verdict and records the ambiguity as a reviewable proposal.

Grounded answer. Only at this point does the LLM write the response, as prose built around numbers that have already been computed and checked. It explains; it does not calculate.

The same flow handles follow-ups such as “compare this with Q2.” The new question goes through the same plan, resolve, retrieve, and validate steps, so the answer is as grounded as the first.

Governance that scales

An architecture like this only works in an enterprise if the right people can make changes the right thing without breaking everything else. PRISM separates its artifacts by who owns them, how often they change, and how far a mistake spreads.

Artifact	Owner	Expertise needed	Change cadence	Blast radius
Prompt rules	App developer	The whole downstream contract: fact binding, calculation, retrieval	Rare: only when the extraction mechanism changes	Every company, every industry, every question
Skills	Domain expert	Filing mechanics for one industry; no code	Occasional: as filing quirks surface	Every company in one industry
Diccionario	End user	Trusts a specific formula for a specific metric	Frequent: incremental	One metric
Conceptos	End user	Their own company’s or vertical’s jargon	Frequent: incremental	One term

The pattern is that the more often something changes, the smaller its blast radius. An analyst who approves a quick-ratio definition affects one metric. A developer who rewrites a prompt rule affects everything, and that is why it changes rarely and sits with the person who understands the whole contract. Governance is distributed, and core application mechanics are decoupled from end-user customization.

Why this belongs on one platform

Every part of this architecture needs a different data structure: documents, vectors, small catalog records, vocabulary maps, and approved definitions. In PRISM, they all live in Couchbase Capella, in one bucket and scope, and are queried with one SQL++ and search surface. Hybrid search needs no second system, and the catalog lookups and chunk retrieval sit next to each other. Embeddings and ingestion come from the Couchbase AI Data Plane, so the whole path from PDF to evidence stays inside one platform.

The lesson goes beyond SEC filings. Whenever answers must be scoped, evidenced, and defensible, such as financial reports, contracts, regulatory submissions, or clinical documents, retrieval alone isn’t enough. Resolve scope first, search with both lexical and semantic signals, let code do the arithmetic, and let humans decide the definitions.

Get started with PRISM

PRISM demonstrates how Couchbase Capella and the Couchbase AI Data Plane can transform complex PDF collections into governed, defensible intelligence. Explore the resources below to see the reference architecture in action, inspect the implementation, and start building on Couchbase Capella today:

• Watch the demo: See the complete setup and live architecture walkthrough in the PRISM Demo Video.
• Explore the code: Access the open-source reference implementation on GitHub.
• Learn about AI Data Plane: Discover how the Couchbase AI Data Plane automates parsing, chunking, and vectorization.

Try Couchbase Capella: Build your own governed search and AI applications with a Couchbase Capella Free Trial.