How to Evaluate a RAG Pipeline With Ragas 0.4.3: Step by Step (2026)

This guide is for engineers who have a working RAG pipeline and no reliable way to tell whether a change made it better or worse. By the end you will have a reviewed test set in JSONL, a Python script that scores every question for faithfulness, context precision, context recall and answer relevancy with Ragas, a GitHub Actions job that fails the pull request when a score drops below its floor, and a realistic idea of what the judge calls cost. Everything here was checked against Ragas 0.4.3 (the latest release, 13 January 2026) on Python 3.12, with the install and the non-LLM metrics actually run on my machine. You need Python 3.11 or newer (the rapidfuzz 3.14.6 pin below requires it), an OpenAI-compatible API key for the judge, and a pipeline you can call as a function. If you want the background on why RAG systems fail in the first place, read the companion post, why RAG fails in production.

Step 1: Install Ragas 0.4.3 with pins that actually import

Ragas pins almost none of its dependencies, and that bites. On a clean install today, pip install ragas==0.4.3 resolves langchain-community 0.4.2, and import ragas then fails because Ragas still imports a Vertex AI chat model that the 0.4 line of langchain-community removed. Pinning langchain-community to the last 0.3 release fixes it. Test set generation also needs rapidfuzz, which is only in the all extra.

# uv 0.13.7; any recent pip works the same way
uv venv -p 3.12 .venv && source .venv/bin/activate
cat > requirements-eval.txt <<'EOF'
ragas==0.4.3
langchain-community==0.3.31   # 0.4.x breaks "import ragas" in 0.4.3
rapidfuzz==3.14.6             # needed by TestsetGenerator
openai==3.3.0
instructor==1.17.0
EOF
uv pip install -r requirements-eval.txt
python -c "import ragas; print(ragas.__version__)"

Expected output:

0.4.3

If you see ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai', the pin did not take. Commit a lock file (uv pip compile requirements-eval.txt -o requirements-eval.lock) so CI resolves the same tree you tested.

One more thing to know before writing code: in 0.4 the top-level evaluate() function still works but emits a DeprecationWarning pointing at the new @experiment API, and the metric docs recommend the metrics in ragas.metrics.collections, which you call directly with ascore(). This guide uses those, so nothing here depends on the deprecated path.

Step 2: Build a test set, then review it by hand

A RAG evaluation is only as good as its questions. Each row needs a question (user_input) and a reference answer (reference); ideally also the IDs of the chunks that contain the answer. Ragas can draft one from your documents using its TestsetGenerator, which the test set generation docs describe:

# evals/make_testset.py  (Ragas 0.4.3)
import json
import openai
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import llm_factory
from ragas.testset import TestsetGenerator

# TextLoader avoids DirectoryLoader's default "unstructured" dependency
docs = DirectoryLoader("docs/", glob="**/*.md", loader_cls=TextLoader).load()
print(f"loaded {len(docs)} documents")

client = openai.OpenAI()
generator = TestsetGenerator(
    llm=llm_factory("gpt-4.1-mini", client=client),
    embedding_model=OpenAIEmbeddings(client=client, model="text-embedding-3-small"),
)
testset = generator.generate_with_langchain_docs(docs, testset_size=50)

with open("evals/testset.draft.jsonl", "w") as fh:
    for row in testset.to_pandas().to_dict(orient="records"):
        fh.write(json.dumps({
            "user_input": row["user_input"],
            "reference": row["reference"],
            "reference_contexts": row["reference_contexts"],
            "synthesizer_name": row["synthesizer_name"],
        }) + "\n")

The docs example wraps a LangChain ChatOpenAI in LangchainLLMWrapper; in 0.4.3 that wrapper is deprecated in favour of llm_factory, which TestsetGenerator accepts, so the script above avoids the warning. Generation calls the LLM many times per document (headline extraction, summaries, then question synthesis), so run it once, not in CI.

Then do the part that matters: open testset.draft.jsonl and edit it. Delete questions no user would ask, fix references that are wrong, and add 20 or 30 real questions from support tickets or search logs, including a few the corpus cannot answer (the right answer is a refusal). A synthetic draft is a starting point; a reviewed set of 50 to 100 rows is your contract. Save it as evals/testset.jsonl:

{"user_input": "How long do refunds take after a return?", "reference": "Refunds are issued within 14 days of the return being received.", "reference_context_ids": ["refund-policy#2"]}
{"user_input": "Can I get a refund on a gift card?", "reference": "No. Gift cards are non-refundable.", "reference_context_ids": ["refund-policy#7"]}

Step 3: Make your pipeline return its contexts

Ragas cannot score retrieval it cannot see. Wrap your pipeline so it returns the generated answer and the exact chunk texts (and IDs) that went into the prompt, after any reranking or truncation:

# my_rag.py: adapt to your stack; the contract is what matters
from typing import List, Tuple

def answer(question: str) -> Tuple[str, List[str]]:
    chunks = retrieve(question, k=5)          # your retriever + reranker
    response = generate(question, chunks)     # your LLM call
    return response, [c.text for c in chunks] # the texts actually in the prompt

def answer_with_ids(question: str) -> Tuple[str, List[str], List[str]]:
    chunks = retrieve(question, k=5)
    return generate(question, chunks), [c.text for c in chunks], [c.id for c in chunks]

Score the contexts as the model saw them, not what the vector store returned before filtering. Otherwise a truncation bug looks like a model bug.

Step 4: Score retrieval without an LLM first

If your test set has reference_context_ids, you can measure retrieval for free and deterministically, before paying a judge. Ragas ships ID-based precision and recall:

# evals/id_metrics.py  (Ragas 0.4.3)
import asyncio
from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision, IDBasedContextRecall

samples = [
    # retriever found both gold chunks, plus one irrelevant chunk
    SingleTurnSample(retrieved_context_ids=["runbook#3", "faq#9", "runbook#4"],
                     reference_context_ids=["runbook#3", "runbook#4"]),
    # retriever missed the gold chunk entirely
    SingleTurnSample(retrieved_context_ids=["faq#1", "faq#2", "blog#7"],
                     reference_context_ids=["policy#12"]),
]

async def main():
    p, r = IDBasedContextPrecision(), IDBasedContextRecall()
    for i, s in enumerate(samples):
        print(i, "precision", round(await p.single_turn_ascore(s), 3),
                 "recall", round(await r.single_turn_ascore(s), 3))

asyncio.run(main())

Real output from my run:

0 precision 0.667 recall 1.0
1 precision 0.0 recall 0.0

These two classes still import from ragas.metrics and print a DeprecationWarning telling you to import them from ragas.metrics.collections instead. In 0.4.3 that import fails with ImportError, so ignore the advice until a later release adds them. Note that ID-based precision is a plain fraction (2 of 3 retrieved were relevant); it does not reward ranking the gold chunk first. The LLM-based ContextPrecision below does.

Step 5: Score faithfulness, context precision, recall and answer relevancy

Now the four LLM-judged metrics. What each one measures, per the Ragas docs:

• Faithfulness: claims in the response supported by the retrieved context, divided by all claims in the response. Needs user_input, response, retrieved_contexts.
• Context precision: whether relevant chunks are ranked above irrelevant ones, as a mean of precision@k at each relevant position (docs). Needs user_input, reference, retrieved_contexts.
• Context recall: the share of claims in the reference answer that can be attributed to the retrieved context (docs). Needs user_input, retrieved_contexts, reference.
• Answer relevancy: the judge writes questions from the response (3 by default, the strictness parameter), and the score is their mean cosine similarity to the real question (docs). Needs user_input, response, and an embedding model.

# evals/run_ragas.py  (Ragas 0.4.3): score the pipeline and fail on regressions
import asyncio, csv, json, math, os, statistics, sys

from openai import AsyncOpenAI
from ragas.cache import DiskCacheBackend
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy, ContextPrecision, ContextRecall, Faithfulness,
)

from my_rag import answer  # answer(q) -> (response, [context strings])

JUDGE_MODEL = os.getenv("RAGAS_JUDGE_MODEL", "gpt-4.1-mini")  # pin it; scores move when it changes
THRESHOLDS = {  # floors for the mean; set them from your own baseline run
    "faithfulness": 0.85,
    "context_recall": 0.80,
    "context_precision": 0.70,
    "answer_relevancy": 0.75,
}
CONCURRENCY = int(os.getenv("RAGAS_CONCURRENCY", "4"))  # stay under your rate limit

client = AsyncOpenAI(max_retries=3)
cache = DiskCacheBackend(cache_dir=".ragas-cache")  # identical judge calls are not re-billed
llm = llm_factory(JUDGE_MODEL, client=client, cache=cache)
emb = OpenAIEmbeddings(client=client, model="text-embedding-3-small", cache=cache)

faithfulness = Faithfulness(llm=llm)
context_precision = ContextPrecision(llm=llm)
context_recall = ContextRecall(llm=llm)
answer_relevancy = AnswerRelevancy(llm=llm, embeddings=emb)


async def score_row(row, sem):
    async with sem:
        response, contexts = await asyncio.to_thread(answer, row["user_input"])
        q, ref = row["user_input"], row["reference"]
        out = {"user_input": q, "response": response, "n_contexts": len(contexts)}
        if not contexts:  # Ragas raises ValueError on empty contexts; count it as a miss
            out.update(faithfulness=0.0, context_precision=0.0, context_recall=0.0)
        else:
            f, p, r = await asyncio.gather(
                faithfulness.ascore(user_input=q, response=response, retrieved_contexts=contexts),
                context_precision.ascore(user_input=q, reference=ref, retrieved_contexts=contexts),
                context_recall.ascore(user_input=q, retrieved_contexts=contexts, reference=ref),
            )
            out.update(faithfulness=f.value, context_precision=p.value, context_recall=r.value)
        a = await answer_relevancy.ascore(user_input=q, response=response)
        out["answer_relevancy"] = a.value
        return out


async def main(path):
    rows = [json.loads(line) for line in open(path) if line.strip()]
    sem = asyncio.Semaphore(CONCURRENCY)
    results = await asyncio.gather(*(score_row(r, sem) for r in rows))

    with open("ragas_results.csv", "w", newline="") as fh:
        w = csv.DictWriter(fh, fieldnames=list(results[0].keys()))
        w.writeheader()
        w.writerows(results)

    failed = False
    for metric, floor in THRESHOLDS.items():
        vals = [r[metric] for r in results if not math.isnan(r[metric])]
        nan = len(results) - len(vals)
        mean = statistics.mean(vals) if vals else 0.0
        ok = mean >= floor
        failed |= not ok
        print(f"{metric:18} mean={mean:.3f} floor={floor:.2f} nan={nan} {'PASS' if ok else 'FAIL'}")
    sys.exit(1 if failed else 0)


if __name__ == "__main__":
    asyncio.run(main(sys.argv[1] if len(sys.argv) > 1 else "evals/testset.jsonl"))

Run it with PYTHONPATH=. python evals/run_ragas.py. It prints one line per metric with PASS or FAIL and writes per-question scores to ragas_results.csv, which is where you will spend your time. I verified the script imports, builds every metric and reaches the API client against Ragas 0.4.3; I did not spend money on a judge for this post, so the floors above are starting points, not results.

Step 6: Interpret the scores

The value of separate metrics is that they point at different components. This is how I read combinations:

Pattern	What it usually means	Where to look
Low context recall	Gold chunk not retrieved	Chunking, embedding model, add BM25, raise k, check filters
High recall, low context precision	Right chunk found but buried in noise	Reranker, lower k, metadata filters
High recall, low faithfulness	Model adds claims not in context	Prompt (answer only from context), temperature, context length
High faithfulness, low answer relevancy	Grounded but off-topic or evasive	Prompt, query rewriting, refusal behaviour
All low on one source	Ingestion problem for that source	Parser, stale index, permissions filter

Two cautions. Faithfulness of 1.0 on a refusal is correct behaviour, so tag unanswerable questions and report them separately. And a mean hides tails: sort the CSV by faithfulness and read the bottom ten answers every time the score moves.

Step 7: Gate pull requests in GitHub Actions

Run the evaluation only when retrieval or prompt code changes, cache the judge responses between runs, and upload the CSV so reviewers can read it. Actions are pinned to commit SHAs (tags in the comments):

# .github/workflows/rag-eval.yml
name: rag-eval
on:
  pull_request:
    paths: ["rag/**", "prompts/**", "evals/**", "requirements-eval.*"]
permissions:
  contents: read
concurrency:
  group: rag-eval-${{ github.ref }}
  cancel-in-progress: true
jobs:
  ragas:
    runs-on: ubuntu-24.04
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
        with:
          version: "0.13.7"
          python-version: "3.12"
          activate-environment: true   # uv pip install needs a venv (or --system)
          enable-cache: true
      - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
        with:
          path: .ragas-cache
          key: ragas-${{ hashFiles('evals/testset.jsonl', 'requirements-eval.lock') }}-${{ github.sha }}
          restore-keys: ragas-${{ hashFiles('evals/testset.jsonl', 'requirements-eval.lock') }}-
      - run: uv pip install -r requirements-eval.lock -r requirements.txt
      - name: Score with Ragas
        env:
          OPENAI_API_KEY: ${{ secrets.RAGAS_JUDGE_KEY }}   # a separate key with a spend cap
          RAGAS_JUDGE_MODEL: gpt-4.1-mini
          PYTHONPATH: .
        run: python evals/run_ragas.py evals/testset.jsonl
      - if: always()
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          name: ragas-results
          path: ragas_results.csv

Pull requests from forks do not receive repository secrets, so this job will fail for external contributors; run it on a label or after merge for those. For the wider pattern of eval gates in CI, see running LLM evals in GitHub Actions with promptfoo.

Step 8: Control LLM-judge cost and bias

Count the calls before you pick a judge. Reading the 0.4.3 metric code: faithfulness makes 2 judge calls per row (extract statements, then verify them), context recall 1, context precision 1 per retrieved chunk, and answer relevancy 3 at the default strictness plus embeddings. With k=5 that is 11 judge calls per question, plus whatever your pipeline itself costs.

At OpenAI's published prices, gpt-4.1-mini is $0.40 per million input tokens and $1.60 per million output, and text-embedding-3-small is $0.02 per million. If you assume around 1,500 input and 200 output tokens per judge call (my assumption; measure yours from the provider dashboard), that is about $0.01 per question, or roughly $1 for 100 questions per run. The disk cache means an unchanged question with an unchanged answer is not re-billed.

Bias is the harder problem. The Ragas paper reported agreement with human annotators of 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance on its WikiEval set, and Zheng et al. documented position, verbosity and self-enhancement bias in LLM judges. What I do about it:

• Hand-label 30 to 50 rows once and check that the judge agrees with you before trusting any threshold.
• Use a different model family for the judge than for generation where you can, to limit self-preference.
• Pin the judge model and Ragas version together; when either changes, re-run the baseline on main and move the floors deliberately.
• Gate on the mean plus a count of rows below, say, 0.5 faithfulness, so one confident hallucination cannot hide in a good average.

Troubleshooting

Symptom	Likely cause	Fix
No module named 'langchain_community.chat_models.vertexai' on import ragas	langchain-community 0.4.x resolved	Pin langchain-community==0.3.31
ImportError: rapidfuzz is required for string distance	Test set generation needs an optional dependency	pip install rapidfuzz or ragas[all]
DeprecationWarning: evaluate() is deprecated	Using the 0.3-style API	Call ragas.metrics.collections metrics with ascore()
LangchainLLMWrapper is deprecated	Following older docs examples	Pass llm_factory("model", client=...) instead
ImportError importing IDBasedContextPrecision from ragas.metrics.collections	The warning text points at a module that does not have it in 0.4.3	Keep importing from ragas.metrics
ValueError: retrieved_contexts is missing	Retriever returned nothing for a row	Score it as 0 explicitly (as in Step 5) and investigate
Metric returns nan	Judge extracted no statements or classifications, for example when the response makes no checkable claim	Count NaNs separately; tag refusal rows
InstructorRetryException: Connection error	Wrong OPENAI_BASE_URL, proxy, or no network in CI	Check the base URL and egress; test with one row
429 rate limit errors mid-run	Too many concurrent judge calls	Lower RAGAS_CONCURRENCY; keep max_retries on the client
Scores jump with no code change	Judge model updated or unpinned	Pin a dated model snapshot; re-baseline on purpose

FAQ

Do I need reference answers to use Ragas?

Not for every metric. Faithfulness and answer relevancy only need the question, the response and the contexts. Context precision and context recall as used here need a reference answer, and they are the metrics that tell you whether retrieval works, so a small reviewed set with references is worth the effort.

How many test questions are enough for a RAG evaluation?

Fifty to a hundred reviewed questions catch most regressions, as long as they cover each document source and include unanswerable ones. With fewer than about 30, a single question flipping moves the mean by several points, so use the per-row CSV rather than the mean to decide.

What is a good faithfulness score?

There is no universal number. Run the suite on your current main branch, read the lowest-scoring answers to check the judge is right, and set the floor a few points below that baseline. Raise it as you fix things.

Is Ragas evaluate() deprecated?

In Ragas 0.4.3, evaluate() still runs but emits a deprecation warning pointing at the @experiment decorator, and the metric docs say the legacy metrics API will be removed in version 1.0. The collections metrics used in this guide are the recommended path.

When I inherit a RAG system, the first thing I check is context recall by source, because a retrieval miss is the most common root cause and the cheapest to fix. In production I keep the reviewed test set in the same repository as the retrieval code, run the suite on every change to chunking, embeddings, filters or prompts, and treat a judge upgrade like a database migration: planned, re-baselined, and reviewed. The official references are the Ragas documentation, the Ragas releases and the original Ragas paper.

Related: How to Build a RAG Pipeline With Postgres and pgvector (2026), MLOps in 2026: Evals Are the New Unit Tests and How to Trace an LLM App with OpenTelemetry and Langfuse (2026).