Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Run test configs in CI

Fail a pull request's CI job when your agent fails a Judgment test config run.

The test runs inside your CI job like any other script. Judgment grades the run, and the SDK raises an error when a test case fails, which fails the step. Whether a failed step blocks the merge is up to your repository’s branch protection.

Write the gate script

Use a pass condition and assert_test=True. With both set, the SDK raises JudgmentTestError when the run doesn’t finish as Completed, or when any test case fails the pass condition. assert_test without a pass condition raises ValueError before anything runs.

import os
import sys
import uuid

from judgeval import Judgeval
from judgeval.exceptions import JudgmentTestError
from lunch_agent import handle_request  # your agent's entry point

JUDGES = ("request-completed", "under-budget")

def lunch_agent(input: str) -> str:
    # One session per test case; handle_request takes (session_id, message).
    return handle_request(str(uuid.uuid4()), input)

def passed(fields, judge_results) -> bool:
    by_name = {r.name: r for r in judge_results}
    return all(
        name in by_name and by_name[name].error is None and by_name[name].value == "Yes"
        for name in JUDGES
    )

client = Judgeval(project_name="lunch-order-agent")
commit = os.environ.get("GITHUB_SHA", "local")[:7]

try:
    result = client.offline_tests.run(
        test_config="lunch-regression",
        agent_function=lunch_agent,
        pass_condition_fn=passed,
        assert_test=True,
        dataset_version=1,
        run_name=f"ci-{commit}",
        timeout_seconds=900,
    )
except JudgmentTestError as err:
    sys.exit(f"Judgment test failed: {err}")

if result is None:
    sys.exit("Project or test config not found")
print(f"Passed: {result.ui_results_url}")

In TypeScript, take the script from Run a test config from your code and add assertTest: true, datasetVersion: 1 and runName to the options of client.offlineTests.run(...).

Watch for these in CI:

  • run() returns None (Python) or null (TypeScript) when the project or test config isn’t found. It doesn’t raise, so check for it, or the job passes without testing anything.
  • TypeScript has no JudgmentTestError. assertTest throws a plain Error, which still fails the step.
  • A timeout raises too. The default wait for judge results is 600 seconds. Size timeout_seconds for your dataset.

Pin dataset_version and, if you want, judge_versions, so the gate measures your change and not a new test case or a new judge version someone saved. Bump them on purpose, in their own pull request.

Add the GitHub Actions job

Store JUDGMENT_API_KEY and JUDGMENT_ORG_ID as repository secrets, along with any keys your agent needs to run, such as its model provider key.

name: Judgment agent test
on:
  pull_request:
    paths:
      - "lunch_agent/**"
      - "tests/judgment_gate.py"
jobs:
  agent-test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "judgeval>=1.3.3" -r requirements.txt
      - run: python tests/judgment_gate.py
        env:
          JUDGMENT_API_KEY: ${{ secrets.JUDGMENT_API_KEY }}
          JUDGMENT_ORG_ID: ${{ secrets.JUDGMENT_ORG_ID }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

For TypeScript, swap the setup steps for actions/setup-node, install judgeval@^1.3.4, and run the gate script with your usual TypeScript runner.

The paths filter keeps the job, and its judge evaluations, off pull requests that don’t touch the agent. Each run’s judge evaluations use Judgment Units; see Billing and plans.

To make it a gate, mark the agent-test job as a required status check in your branch protection rules, and remove the paths filter first. When paths skips the workflow, GitHub never reports the check, so a required check stays pending and the pull request can’t merge. If you’d rather keep paths, leave the check optional.

When all-or-nothing is too strict

Judges are models, so a borderline test case can flip between runs. assert_test fails the job on any single failed case. If you’d rather pass at 9 out of 10, skip assert_test, keep the pass condition, and replace the try block in the gate script with your own check:

result = client.offline_tests.run(
    test_config="lunch-regression",
    agent_function=lunch_agent,
    pass_condition_fn=passed,
    dataset_version=1,
)
if result is None or result.status != "completed":
    sys.exit("Run did not complete")
case_passed = [
    bool(r.scorers_data) and all(s.success for s in r.scorers_data)
    for r in result.results
]
if sum(case_passed) < 9:
    sys.exit(f"Only {sum(case_passed)} of {len(case_passed)} cases passed: {result.ui_results_url}")

The run page in Judgment shows which cases failed and the judges’ reasoning; see Read and compare test config runs.

Was this page helpful?