How to Run LLM Evals in CI with GitHub Actions (promptfoo, 2026)

This guide is for engineers who already have an LLM feature in production and want prompt or model changes to go through the same pull request gate as code. By the end you will have a GitHub Actions workflow that runs a promptfoo eval suite on every PR that touches a prompt, combines deterministic checks with an LLM-as-judge rubric, fails the build below a pass-rate threshold, caches responses so reruns cost nothing, and posts a summary comment on the PR. There is a DeepEval version for teams that prefer pytest. Tested on 20 October 2026 with promptfoo 0.124.1 (which requires Node 22.22 or newer), promptfoo-action v1 (1.3.4), DeepEval 4.3.1 and actions/checkout@v7. You need a repository, an API key for your model provider stored as a GitHub secret, and Node 24 locally.

One ownership note before you commit to a tool: promptfoo announced on 9 March 2026 that it was joining OpenAI. The README says it remains open source and MIT licensed, and it still supports Anthropic, Google and local models, which is what this guide uses. If you would rather not have an eval harness owned by a model vendor, the DeepEval section gives you an Apache-2.0 alternative with the same structure. For the bigger picture on why prompts deserve a release process at all, see the companion piece on prompt management in 2026.

Step 1: Lay out prompts and evals in the repository

Keep the prompt text in its own file so a diff shows exactly what changed, and keep the eval suite next to it. The layout below is what the rest of the guide assumes.

.
├── prompts/
│   └── triage.txt                 # the prompt under test
├── evals/
│   ├── promptfooconfig.yaml       # providers, judge, shared assertions
│   └── tests/
│       └── triage.yaml            # golden cases, one per behaviour
└── .github/workflows/
    └── llm-evals.yml

The prompt itself asks for structured output, because structured output is what makes cheap deterministic checks possible:

You are a support triage assistant.
Classify the ticket into exactly one category: BILLING, ACCOUNT or OTHER.
Reply with JSON only, in the form {"category": "...", "summary": "..."}.
The summary is one neutral sentence of at most 25 words.

Ticket: {{ticket}}

Promptfoo fills {{ticket}} from each test's vars. Start with 20 to 50 golden cases taken from real traffic (with personal data removed), each one describing a behaviour you would be upset to lose.

Step 2: Write deterministic assertions first

Deterministic assertions cost nothing beyond the model call, run in milliseconds and never flake. Put the ones that apply to every case in defaultTest so each test only states what is specific to it. Promptfoo's assertion reference lists the built-in types: is-json, contains, regex, javascript, python, latency, cost, tool-call checks and many more.

# evals/promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: support-triage prompt regression suite

prompts:
  - file://../prompts/triage.txt   # resolved relative to this file

providers:
  # The model your app actually calls. Pin a dated snapshot, not an alias.
  - id: anthropic:messages:claude-haiku-4-5-20251001
    config:
      temperature: 0
      max_tokens: 300

defaultTest:
  options:
    # Judge for llm-rubric assertions (Step 3). Pinned, temperature 0,
    # and a different model from the one under test.
    provider:
      id: anthropic:messages:claude-sonnet-4-5-20250929
      config:
        temperature: 0
  assert:
    # Applied to every test: cheap, deterministic, never flaky.
    - type: is-json
    - type: javascript
      value: "['BILLING','ACCOUNT','OTHER'].includes(JSON.parse(output).category)"
    - type: latency
      threshold: 8000        # milliseconds
    - type: cost
      threshold: 0.002       # dollars per call; catches a prompt that tripled in length

tests: file://tests/triage.yaml

# evals/tests/triage.yaml
- description: double charge goes to billing
  vars:
    ticket: "I was charged twice for my October invoice, please refund one."
  assert:
    - type: javascript
      value: "JSON.parse(output).category === 'BILLING'"

- description: password reset goes to account
  vars:
    ticket: "The password reset email never arrives, I checked spam."
  assert:
    - type: javascript
      value: "JSON.parse(output).category === 'ACCOUNT'"

Validate the file before spending a single token:

npx -y promptfoo@0.124.1 validate -c evals/promptfooconfig.yaml

Configuration is valid.

Step 3: Add LLM-as-judge assertions with explicit thresholds

Some behaviours (tone, faithfulness to a source, whether a summary is actually a summary) cannot be checked with a regex. That is what model-graded assertions are for. In promptfoo the workhorse is llm-rubric; g-eval, factuality, context-faithfulness and answer-relevance are also available.

# append to evals/tests/triage.yaml
- description: angry customer gets a polite summary
  vars:
    ticket: "This is the third time your app logged me out. Fix it or I cancel."
  assert:
    - type: llm-rubric
      value: >-
        The summary field is neutral and polite, does not repeat insults,
        and mentions repeated logouts.
      threshold: 0.8     # judge score must reach 0.8, not just "pass"
      metric: tone       # groups results in the report

Three rules keep the judge honest. Pin it: the model-graded docs let you set the grader once in defaultTest.options.provider or per assertion, and a per-assertion provider overrides the default without inheriting its config, so repeat the whole object if you override. Use a different model from the one under test, so the judge is not grading its own habits. And set a threshold: without one, an explicit pass from the judge decides, and with one the numeric score must meet it. Judges disagree with human experts often enough that I would never gate on one without a threshold and a repeat count (the MLOps evals post has the numbers on judge agreement).

Step 4: Run the suite locally and understand the exit codes

Export the key and run the eval. -o writes machine-readable results you can inspect or upload as an artifact.

export ANTHROPIC_API_KEY=...        # never commit this
npx -y promptfoo@0.124.1 eval -c evals/promptfooconfig.yaml \
  --no-share --no-progress-bar -o results.json -o results.junit.xml
echo "exit=$?"

The exit code is the contract with CI. Per the command-line docs, eval returns 100 when at least one test fails, 1 for any other error, and 0 otherwise. PROMPTFOO_PASS_RATE_THRESHOLD relaxes the default of 100 percent. I checked this on a two-test suite with one deliberate failure, using promptfoo's offline echo provider so it cost nothing:

Results:
  ✓ 1 passed (50.00%)
  ✗ 1 failed (50.00%)
  0 errors (0%)
Duration: 0s (concurrency: 4)

Writing output to results.json
exit=100

Rerunning the same suite with PROMPTFOO_PASS_RATE_THRESHOLD=50 exited 0. The summary counts live in results.json:

jq '.results.stats | {successes, failures, errors}' results.json

{
  "successes": 1,
  "failures": 1,
  "errors": 0
}

promptfoo view opens a local web UI with every output, score and judge explanation side by side. Use it while writing tests; CI only needs the exit code.

Step 5: Add the GitHub Actions workflow with PR comments

The official promptfoo-action runs npx promptfoo@<version>, passes changed prompt files through --prompts, and posts a pass/fail comment on the PR. Its inputs include fail-on-threshold (required pass percentage), repeat and repeat-min-pass (for flaky tests), cache-path and max-concurrency.

# .github/workflows/llm-evals.yml
name: llm-evals

on:
  pull_request:
    paths:                       # only spend tokens when prompts or evals change
      - "prompts/**"
      - "evals/**"

concurrency:                     # a new push cancels the old, half-finished eval
  group: llm-evals-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read                 # checkout
  pull-requests: write           # PR comment

jobs:
  evals:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7

      - uses: actions/setup-node@v7
        with:
          node-version: "24"     # promptfoo 0.124.x needs Node >= 22.22.0

      - name: Restore promptfoo response cache
        uses: actions/cache@v6
        with:
          path: .promptfoo-cache
          key: promptfoo-${{ runner.os }}-${{ hashFiles('prompts/**', 'evals/**') }}
          restore-keys: |
            promptfoo-${{ runner.os }}-

      - name: Run evals
        uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          config: evals/promptfooconfig.yaml
          prompts: |
            prompts/**
          use-config-prompts: true   # globs only drive change detection; evaluate config prompts
          promptfoo-version: 0.124.1 # pin; "latest" changes under you
          cache-path: .promptfoo-cache
          fail-on-threshold: 95      # suite pass rate required to go green
          max-concurrency: 4
          no-share: true             # keep outputs off promptfoo's share server

Then make the evals job a required status check in branch protection, otherwise a red eval is just a suggestion. If you would rather not use the action, the same thing works as a plain run: step calling npx promptfoo@0.124.1 eval with PROMPTFOO_PASS_RATE_THRESHOLD=95, and the exit code fails the job.

Secrets are the sharp edge. Workflows triggered by pull_request from a fork do not receive repository secrets, so the eval will fail on contributor PRs. Do not "fix" this with pull_request_target and a checkout of the PR head: that runs untrusted code with your API key in the environment, and the action's README warns against exactly that. Run evals on branches inside the repo, or have a maintainer re-run them. The GitHub Actions supply-chain post covers why this matters.

Step 6: Control cost and runtime

An eval suite that costs real money on every push gets disabled within a month. These are the levers, roughly in order of payoff.

1. Path filters and concurrency. The paths and concurrency blocks above mean evals run only when prompts change and only for the latest commit.
2. The response cache. Promptfoo caches provider responses on disk by default; persisting cache-path with actions/cache means a rerun of an unchanged prompt and test makes no API calls. The CI/CD docs also document PROMPTFOO_CACHE_PATH and PROMPTFOO_CACHE_TTL (default example: 86400 seconds).
3. Rerun only failures. After a red run, --filter-failing results.json reruns only tests that failed or errored, which is what you want while iterating on a fix.
4. Cheap model under test, stronger judge only where needed. Deterministic checks on every case, llm-rubric only on the cases that need it.
5. Cost assertions. The cost assertion in defaultTest fails any test whose single call exceeds your budget, which also catches prompts that quietly tripled in length.
6. Tiered suites. A 30-case smoke suite on every PR, the full suite nightly on main via schedule:.

If your team routes model calls through a gateway, point promptfoo at it and you get per-key budgets and spend reports for CI for free; the LiteLLM proxy guide shows that setup.

Step 7: The DeepEval alternative (pytest style)

DeepEval (Apache-2.0, 4.3.1 on PyPI, Python 3.9 or newer) expresses the same idea as pytest tests: build an LLMTestCase, attach metrics with thresholds, call assert_test. Note that DeepEval 4 renamed LLMTestCaseParams to SingleTurnParams; the old name still imports with a deprecation warning.

# tests/test_triage_llm.py
# pip install deepeval==4.3.1 anthropic
import json, pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, SingleTurnParams
from deepeval.metrics import GEval
from deepeval.models import AnthropicModel
from app.triage import run_triage   # your application code

judge = AnthropicModel(model="claude-sonnet-4-5-20250929", temperature=0)

tone = GEval(
    name="Polite summary",
    criteria="The summary is neutral and polite and does not repeat insults.",
    evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT],
    threshold=0.8,
    model=judge,
)

CASES = [
    ("I was charged twice for my October invoice.", "BILLING"),
    ("This is the third time your app logged me out. Fix it or I cancel.", "ACCOUNT"),
]

@pytest.mark.parametrize("ticket,expected", CASES)
def test_triage(ticket, expected):
    output = run_triage(ticket)
    assert json.loads(output)["category"] == expected   # deterministic first
    assert_test(LLMTestCase(input=ticket, actual_output=output), [tone])

The CLI flags map neatly onto the CI concerns above. From the deepeval test run source: -n runs tests in parallel processes, -c uses the local cache, -r repeats each test, -i ignores metric errors and -x stops at the first failure.

      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install deepeval==4.3.1 anthropic -r requirements.txt
      - name: Run DeepEval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: deepeval test run tests/test_triage_llm.py -n 4 -c

Pick promptfoo if your prompts are mostly config and you want YAML, a web viewer and PR comments out of the box. Pick DeepEval if your eval cases need to call real application code and your team already lives in pytest.

Step 8: Make flaky evals deterministic enough to gate on

Even at temperature 0, model outputs and judge scores vary between runs. A gate that goes red one time in ten will be ignored by week two. Handle it in layers.

      - name: Run evals
        uses: promptfoo/promptfoo-action@v1
        with:
          # ...inputs from Step 5...
          repeat: 3              # run each test three times
          repeat-min-pass: 2     # a test passes if at least 2 of 3 runs pass
          fail-on-threshold: 95

Move anything checkable by code out of the judge. Tighten rubric wording until two engineers reading it would agree on a verdict. Drop judge thresholds slightly below the score your current prompt reliably gets, then ratchet up. Separate errors (timeouts, rate limits, malformed judge replies) from failures when you read results.json, because a rate limit is not a regression.

Troubleshooting

Symptom	Likely cause	Fix
npx promptfoo fails with an engine or syntax error on the runner	promptfoo 0.124.x declares node >=22.22.0	Add actions/setup-node@v7 with node-version: "24" before the eval step
Job fails with exit code 100 but no assertion looks wrong	Pass rate threshold defaults to 100 percent	Set fail-on-threshold (action) or PROMPTFOO_PASS_RATE_THRESHOLD (CLI) to the rate you actually require
Evals fail only on PRs from forks, with 401 or missing key errors	pull_request from forks gets no repository secrets	Run evals on in-repo branches or via maintainer re-run; do not switch to pull_request_target with a PR-head checkout
No PR comment appears	Missing pull-requests: write permission, or disable-comment set	Add the permission at workflow or job level and check the action inputs
Same test passes and fails on reruns of identical code	Judge variance or temperature above 0	Pin the judge, set temperature 0 where supported, use repeat: 3 with repeat-min-pass: 2, move checks into code
Every rerun costs the same as the first run	Cache directory not persisted, or no-cache: true	Persist cache-path with actions/cache@v6; key it on a hash of prompts and evals
llm-rubric errors with no score	Judge reply truncated or malformed	Raise the judge's max_tokens; promptfoo treats a reply with no verdict and no score as an error
Action says no relevant files and skips	Change detection found no file matching prompts globs	Fix the glob (it resolves from working-directory) or set force-run: true
Many errors (not failures) in results.json	Provider rate limits or timeouts	Lower max-concurrency, add --delay, or route through a gateway with retries
DeepEval warns LLMTestCaseParams is deprecated	DeepEval 4 renamed it	Import SingleTurnParams from deepeval.test_case

FAQ

What is the difference between promptfoo and DeepEval?

Both run LLM test suites and return a failing exit code in CI. Promptfoo is a Node CLI driven by YAML config, with a web viewer, red-teaming and an official GitHub Action that comments on PRs; it is MIT licensed and owned by OpenAI since March 2026. DeepEval is a Python framework built on pytest, Apache-2.0 licensed, and fits best when eval cases need to call your application code.

How many test cases do I need for LLM evals in CI?

Start with 20 to 50 golden cases from real traffic that cover the behaviours you care about most, plus one regression case for every bug you fix. Keep the per-PR suite small enough to finish in a few minutes and run the full set nightly.

Should LLM-as-judge evals block a merge?

Only with a pinned judge model, an explicit score threshold and repeats, and ideally after a few weeks of running in report-only mode so you know the baseline. Deterministic assertions can block from day one.

How much do LLM evals in CI cost?

It depends on suite size, models and how often prompts change, so measure it: promptfoo's cost assertion and the tokenUsage block in results.json give per-run figures. Path filters, caching and rerunning only failures usually make the per-PR cost small.

When I set this up, the first thing I check is that the eval job is a required status check, because an optional red eval is decoration. The second is that every test that can be a javascript assertion is one, so the judge only sees what genuinely needs judgement. Then I watch the errors count for a week before raising the threshold. References: promptfoo CI/CD docs, promptfoo assertions, promptfoo-action, and DeepEval CI/CD docs.

Related: MLOps in 2026: Evals Are the New Unit Tests, Judges Agree With Experts 66% of the Time, and OpenAI's Evals API Shuts Down on 30 November, CI/CD in 2026: Agents Opened 1M PRs in 5 Months, Bots Write 1 in 5 Reviews, and Most Agent PRs Get No Human Look and Prompt Management in 2026: Registries, Rollbacks and Five Acquisitions.