Evaluating RAG Pipeline Quality With RAGAS

A RAG pipeline can fail in two completely different places, and most teams have no clean way to tell which one broke. RAGAS solves that by scoring retrieval and generation separately, without needing a human-labeled answer key for every question. That distinction, retrieval versus generation, is the whole point of this piece.

What RAGAS is and what problem it was designed to solve

RAGAS stands for Retrieval Augmented Generation Assessment. Shahul Es and Jithin James, working at Exploding Gradients, built it with Luis Espinosa-Anke and Steven Schockaert from Cardiff University, and the paper landed at EACL 2024. The design goal was narrow and practical: evaluate a RAG system without requiring a stack of human-annotated reference answers.

That reference-answer requirement is the real bottleneck for most teams trying to evaluate RAG quality. Writing ground-truth answers for hundreds of domain-specific queries takes weeks, and by the time it's done, the corpus has usually changed. RAGAS sidesteps most of that by scoring retrieval and generation as separate components, so a team gets a diagnosis, not just a grade.

It plugs into LangChain and LlamaIndex natively, which matters because those are the two frameworks most RAG pipelines are already built on. Low friction, in other words: no rearchitecting required to start scoring.

The library has grown a lot since its first release. Version 0.2 expanded its scope past RAG specifically, into evaluation for any LLM application, including agents. Version 0.4, released in late 2025, introduced an experiment-based architecture with a collections API for organizing evaluation runs. By 2025 the library had shipped its 50th metric. But for RAG pipelines specifically, the four original core metrics are still the ones doing the real diagnostic work: context precision, context recall, faithfulness, and answer relevancy.

Two of those four score retrieval. Precision asks whether the chunks that got retrieved were actually relevant. Recall asks whether all the relevant chunks in the corpus got retrieved at all. Low precision usually looks like retrieved chunks that are topically nearby but don't actually answer the question, pulled in despite not being genuinely relevant. The fix is often cheaper than it sounds: cleaning up the retrieval set frequently pushes faithfulness scores up too, with no change to the generator at all. Improving precision, in that sense, is often the highest-leverage fix available.

Recall failures come from a different place. Maybe top-k is set too small and a relevant passage falls just outside the retrieval window. Maybe the information exists in the corpus but got chunked in a way that makes it unfindable. Queries that require connecting facts across multiple documents are especially punishing here: if retrieval only surfaces part of the needed evidence, the model was never given what it needed to answer correctly. That's the core diagnostic principle underneath all of this: a model can't generate what it never saw. Blaming the generator for a gap that started at retrieval means fixing the wrong thing.

Precision and recall pull against each other. Widening top-k to catch more relevant passages usually drags in more noise too, so recall climbs while precision slides. Watched together, the two numbers show whether a team is navigating that trade-off deliberately or just cranking one dial and hoping. In domain-heavy corpora, financial filings, semiconductor manufacturing documentation, recall matters even more, because the relevant evidence is often sparse to begin with, and missing it is expensive.

Faithfulness is the third core metric, and it's the one that catches hallucination directly: does the generated answer actually stick to what was retrieved? A low faithfulness score sitting next to a high-precision retrieval set is about as clear a signal as this framework produces. The right information was there, and the model wandered off from it anyway. That's a generation problem, not a retrieval problem, and no amount of re-indexing the corpus will fix it.

Faithfulness catches two recognizable patterns. One is a model tacking on a plausible-sounding fact that isn't in any retrieved chunk. The other is subtler: the model synthesizes across several retrieved passages in a way that produces a claim none of them individually supports. Neither shows up if you're only looking at whether the answer sounds confident.

Faithfulness is the one metric among the core four that can be harder to compute without some grounding reference. That's exactly the gap RAGAS's synthetic test data generation is built to fill, covered further down.

A faithfulness score above 0.8 is the general bar for production readiness in most deployment contexts. Worth flagging though: comparisons across RAGAS Faithfulness, DeepEval's HallucinationMetric, and LLM-as-Judge Groundedness show the three can diverge sharply on abstention responses. A model that refuses to answer rather than guess can score well on Faithfulness, since it made no unsupported claims, while getting penalized by a different evaluator's rubric for not answering at all. Scores near that boundary need a second look before anyone trusts them at face value.

Answer Relevancy: whether the response actually answers the question that was asked

Answer Relevancy works differently from the other three, and it's worth separating out. RAGAS computes it by comparing the generated answer against the original question to assess how directly the response addresses what was actually asked. The relevance score reflects how closely the response aligns with what was actually asked.

What it's checking isn't factual accuracy. It's whether the response is actually on-target. A completely grounded, fully faithful answer can still score poorly here if it doesn't address what the user actually asked, which is a distinction the other three metrics can't make on their own.

Low Answer Relevancy tends to show up in a few recognizable shapes:

• The answer responds to a related but subtly different question, often because the retriever pulled in adjacent-topic chunks that nudged the generation off course
• The direct answer is there, but it's buried under caveats or background context
• The prompt template itself encourages a verbose style that dilutes whatever on-topic content exists

That last one is worth sitting with. It's a prompt engineering problem, not a retrieval problem and not a hallucination problem, and Answer Relevancy is the only one of the four metrics built to catch it.

How the four metrics combine into a diagnostic picture, not just a composite score

None of these four numbers means much in isolation. The value is in reading them together, because specific combinations point to specific fixes.

Low precision paired with low faithfulness suggests retrieval and generation are both struggling, often in connected ways. Fix retrieval first here; touching the prompt or the generator won't help while it's still being handed poor context. Meanwhile, high faithfulness sitting next to low Answer Relevancy tells a different story: the grounding is fine, the model isn't making anything up, but the response still isn't serving what the user actually wanted. That points to a generation or prompt problem, not a hallucination problem.

This is what makes RAGAS more useful than a single quality score ever could be. Each dimension maps to a specific corrective action, so a team isn't left guessing whether a bad output means "fix the retriever" or "fix the model." Running all four together also guards against a mistake that's easy to make when optimizing on a single number: pushing recall up through retrieval changes until precision quietly collapses, and not noticing because the one metric being watched went in the right direction.

Extended metrics for pipelines that need finer-grained diagnostics

The core four cover the primary failure modes in a standard RAG pipeline. But some architectures need more granular instruments, and RAGAS has added several as the library expanded.

Noise Sensitivity measures how often the system produces an incorrect response when working from a mix of relevant and irrelevant retrieved documents. It's scored from 0 to 1, lower is better, and it's the metric to reach for when the retriever is known to pull mixed-relevance chunks and the open question is whether the generator handles that noise gracefully or amplifies it into a wrong answer.

Tool Call Accuracy matters for agentic workflows, where RAG is one capability among several a model can choose to invoke. It checks whether the agent picked the right tool at the right step, which becomes relevant the moment retrieval stops being automatic and becomes a decision the model makes on its own.

The DAG metric structures evaluation as a directed acyclic graph rather than a flat score. It's deterministic and customizable, letting teams define branching evaluation logic for pipelines too complex for a single linear metric to capture.

None of this is where a team should start, though. Extended metrics earn their cost, in judge calls and added complexity, only after the core four have been improved and a specific failure mode remains that they don't explain.

Generating test data without a labeled dataset

Manually annotated evaluation datasets are slow, expensive, and, for most domain-specific corpora, simply don't exist. That's the gap RAGAS's synthetic test data generation is built to close: it generates test questions straight from the document corpus being evaluated, no separate annotation project required.

The mechanism behind it is a knowledge graph. RAGAS builds a graph out of the corpus and traverses it to generate questions, rather than sampling passages at random and writing questions off them in isolation. That graph structure is what makes multi-hop question generation possible in the first place.

Question generation follows an evolutionary paradigm, borrowing from the Evol-Instruct approach, where question complexity and type are systematically varied instead of left to whatever an LLM naturally produces. Left alone, LLMs cluster around the same few obvious question shapes. RAGAS forces broader coverage through three synthesizers:

• SingleHopSpecificQuerySynthesizer, roughly 50% of generated questions: specific, factual, answerable from a single document
• MultiHopAbstractQuerySynthesizer, roughly 25%: abstract reasoning that spans multiple documents
• MultiHopSpecificQuerySynthesizer, the remaining share: specific factual questions that still require pulling information from more than one source

The multi-hop cases are deliberately built to require genuine cross-document reasoning. Plenty of existing multi-hop benchmarks have a shortcut problem, where the answer turns out to be findable without actually connecting the two documents the question was supposed to require. RAGAS's generation approach is built to avoid producing that kind of false multi-hop question.

The practical effect: a team can generate a meaningful evaluation set from its own corpus in hours. That removes the annotation bottleneck that, historically, has pushed real RAG evaluation until after a system is already in production, which is the worst time to discover a retrieval problem.

The LLM-as-judge mechanism and its known failure modes

Every metric described so far depends on an LLM acting as judge, scoring faithfulness, relevance, and precision at a volume no human annotation process could match. That scale is the entire value proposition. It's also where the known failure modes live, and they're worth naming plainly rather than glossing over.

Verbosity inflation is the most common: judges tend to rate longer, more detailed answers higher, independent of whether that length adds accuracy. Self-preference shows up when a judge model favors outputs that stylistically resemble its own generation patterns over ones that don't. Position effects are the third: in a list or a side-by-side comparison, earlier items tend to get rated more favorably regardless of content.

There's also a practical constraint worth being blunt about. Most LLM-as-judge setups run on frontier models like GPT-4, which means multi-second latency, real API cost per evaluation call, prompt sensitivity, and run-to-run inconsistency. That combination rules them out as a real-time production guardrail. RAGAS is built for batch evaluation, run against a test set or a sampled slice of production traffic, not for scoring every live response inline before it reaches a user.

And the abstention problem raised earlier deserves a second mention here, because it's a judge-mechanism issue as much as a metric-design one. Comparisons of RAGAS Faithfulness, DeepEval, and LLM-as-Judge Groundedness have found meaningful disagreement on refusal responses: the identical abstention can register as perfectly faithful under one framework and get marked down under another. Any team leaning on these scores near a decision threshold needs to know which judge is making the call, and what it does with a model that simply says it doesn't know.