---
title: "Run test configs in CI"
deprecated: true
description: "Fail a pull request's CI job when your agent fails a Judgment test config run."
sidebar:
  label: "Run test configs in CI"
seo:
  title: "Run test configs in CI | Judgment Docs"
  description: "Fail a pull request's CI job when your agent fails a Judgment test config run."
---

> **Test configs are deprecated**
>
> This page gates CI on a test config run (`client.offline_tests`), which the
> SDK and API allow only for organizations that used test configs before test
> suites launched. For test suites, see
> [Fail a CI job on a failed case](/documentation/tests/run-a-test-suite#fail-a-ci-job-on-a-failed-case).

The test runs inside your CI job like any other script. Judgment grades the run, and the SDK raises an error when a test case fails, which fails the step. Whether a failed step blocks the merge is up to your repository's branch protection.

## Write the gate script

Use a pass condition and `assert_test=True`. With both set, the SDK raises `JudgmentTestError` when the run doesn't finish as **Completed**, or when any test case fails the pass condition. `assert_test` without a pass condition raises `ValueError` before anything runs.

```python tests/judgment_gate.py
import os
import sys
import uuid

from judgeval import Judgeval
from judgeval.exceptions import JudgmentTestError
from lunch_agent import handle_request  # your agent's entry point

JUDGES = ("request-completed", "under-budget")

def lunch_agent(input: str) -> str:
    # One session per test case; handle_request takes (session_id, message).
    return handle_request(str(uuid.uuid4()), input)

def passed(fields, judge_results) -> bool:
    by_name = {r.name: r for r in judge_results}
    return all(
        name in by_name and by_name[name].error is None and by_name[name].value == "Yes"
        for name in JUDGES
    )

client = Judgeval(project_name="lunch-order-agent")
commit = os.environ.get("GITHUB_SHA", "local")[:7]

try:
    result = client.offline_tests.run(
        test_config="lunch-regression",
        agent_function=lunch_agent,
        pass_condition_fn=passed,
        assert_test=True,
        dataset_version=1,
        run_name=f"ci-{commit}",
        timeout_seconds=900,
    )
except JudgmentTestError as err:
    sys.exit(f"Judgment test failed: {err}")

if result is None:
    sys.exit("Project or test config not found")
print(f"Passed: {result.ui_results_url}")
```

In TypeScript, take the script from [Run a test config from your code](/documentation/tests/run-a-test-from-your-code) and add `assertTest: true`, `datasetVersion: 1` and `runName` to the options of `client.offlineTests.run(...)`.

Watch for these in CI:

- `run()` returns `None` (Python) or `null` (TypeScript) when the project or test config isn't found. It doesn't raise, so check for it, or the job passes without testing anything.
- TypeScript has no `JudgmentTestError`. `assertTest` throws a plain `Error`, which still fails the step.
- A timeout raises too. The default wait for judge results is 600 seconds. Size `timeout_seconds` for your dataset.

Pin `dataset_version` and, if you want, `judge_versions`, so the gate measures your change and not a new test case or a new judge version someone saved. Bump them on purpose, in their own pull request.

## Add the GitHub Actions job

Store `JUDGMENT_API_KEY` and `JUDGMENT_ORG_ID` as repository secrets, along with any keys your agent needs to run, such as its model provider key.

```yaml .github/workflows/judgment-test.yml
name: Judgment agent test
on:
  pull_request:
    paths:
      - "lunch_agent/**"
      - "tests/judgment_gate.py"
jobs:
  agent-test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "judgeval>=1.3.3" -r requirements.txt
      - run: python tests/judgment_gate.py
        env:
          JUDGMENT_API_KEY: ${{ secrets.JUDGMENT_API_KEY }}
          JUDGMENT_ORG_ID: ${{ secrets.JUDGMENT_ORG_ID }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```

For TypeScript, swap the setup steps for `actions/setup-node`, install `judgeval@^1.3.4`, and run the gate script with your usual TypeScript runner.

The `paths` filter keeps the job, and its judge evaluations, off pull requests that don't touch the agent. Each run's judge evaluations use Judgment Units; see [Billing and plans](/documentation/administration/billing-and-plans).

To make it a gate, mark the `agent-test` job as a required status check in your branch protection rules, and remove the `paths` filter first. When `paths` skips the workflow, GitHub never reports the check, so a required check stays pending and the pull request can't merge. If you'd rather keep `paths`, leave the check optional.

## When all-or-nothing is too strict

Judges are models, so a borderline test case can flip between runs. `assert_test` fails the job on any single failed case. If you'd rather pass at 9 out of 10, skip `assert_test`, keep the pass condition, and replace the `try` block in the gate script with your own check:

```python
result = client.offline_tests.run(
    test_config="lunch-regression",
    agent_function=lunch_agent,
    pass_condition_fn=passed,
    dataset_version=1,
)
if result is None or result.status != "completed":
    sys.exit("Run did not complete")
case_passed = [
    bool(r.scorers_data) and all(s.success for s in r.scorers_data)
    for r in result.results
]
if sum(case_passed) < 9:
    sys.exit(f"Only {sum(case_passed)} of {len(case_passed)} cases passed: {result.ui_results_url}")
```

The run page in Judgment shows which cases failed and the judges' reasoning; see [Read and compare test config runs](/documentation/tests/compare-runs).