Run your agent on a test suite

Start a run, run your agent once per case, and read each case's result.

A run has four steps: start the run, run your agent once per case, wait for the cases to be scored, and read the results. Judgment does not call your agent; your code runs it.

Prerequisites: a test suite with at least one published version, and your API key, organization ID, and project name (see Authentication).

Start a run and run your agent

Start a run, then run your agent once per case with judgment.test.run_id and judgment.test.case_id on the case’s root span. The examples use the Tracer.test_case() wrapper; setting the two attributes directly is equivalent (see Connect your agent).

import os

from judgeval import Judgeval, Tracer

client = Judgeval(project_name="support-agent")
run = client.test_runs.start(
    TEST_SUITE_ID, candidate=os.environ["GIT_SHA"], trigger="ci"
)

for case in run["cases"]:
    with Tracer.test_case(run["run_id"], case["case_id"]):
        support_agent(case["input"])

import { Judgeval, Tracer } from "judgeval";

const judgeval = await Judgeval.create({ projectName: "support-agent" });
const run = await judgeval.testRuns.start(TEST_SUITE_ID, {
    candidate: process.env.GIT_SHA,
    trigger: "ci",
});

for (const c of run.cases) {
    await Tracer.testCase(run.run_id, c.case_id, () => supportAgent(c.input));
}

The run uses the test suite’s latest published version. Each case in run["cases"] has its case_id, the input to send to your agent, and the tagging headers for an agent you call over HTTP. Judgment scores each case as soon as its trace arrives.

start accepts the following optional parameters:

Read results

client.test_runs.get(run_id) returns the run’s counts:

The run is finished when cases_scored equals cases_total. client.test_runs.list(test_suite_id) lists a test suite’s runs, newest first.

client.test_runs.results(run_id) returns one entry per case, with one entry per check:

{
  "case_id": "…", "trace_id": "…", "scored_at": "…", "passed": false,
  "checks": [{
    "judge_id": "…", "judge_major_version": 1, "judge_minor_version": 0,
    "op": "eq", "expected": "Yes", "judge_output": "No", "passed": false,
    "reason": "The final answer does not mention a refund. <citation …>",
    "error_message": null
  }]
}

Fail a CI job on a failed case

To use a run as a CI gate, wait until every case is scored and exit with an error when any case did not pass:

import sys
import time

Tracer.force_flush()
while (summary := client.test_runs.get(run["run_id"]))["cases_scored"] < summary["cases_total"]:
    time.sleep(5)
if summary["cases_passed"] < summary["cases_total"]:
    sys.exit(f"{summary['cases_total'] - summary['cases_passed']} test cases failed")

A run has no deadline, so give the CI job a timeout in case a trace never arrives (see a case is never scored). Store JUDGMENT_API_KEY and JUDGMENT_ORG_ID as CI secrets.

Retry failures

Start a new run with {"kind": "last_failures", "run_id": "<earlier run>"} as its case_filter. It runs only the cases that failed or were never scored in that run.

Score an existing run

After you change a test suite’s cases and publish a new version, you can score the existing traces again without rerunning your agent:

new_run = client.test_runs.score(run_id)

This creates a new run, scored with the test suite’s latest published version, from the cases that have a trace and whose input did not change. The original run is not modified. The new run’s cases are scored in the background, so poll client.test_runs.get(new_run["run_id"]) for progress. When the run already used the latest version, scoring it again retries cases that were never scored.

Troubleshooting: a case is never scored

A run has no deadline, so a case waits until its trace arrives. If cases_with_output stays below cases_total, check that:

• The attributes are on the root span, not a child span.
• The trace lands in the test suite’s project.
• Only baggage is sent to a deployed agent. A traceparent from a span Judgment never receives leaves the case without a root span.
• Each case runs as its own trace. Judgment scores the first trace for each case and ignores later traces.

If a check fails with error_message: "Judge not found", its judge was deleted after the case was created. A check can also fail with rate limited or usage limit reached when your organization hits a limit.

Use the HTTP API

Every SDK call maps to a route under https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites, called with your API key and organization ID:

Starting and scoring runs need the Developer role or higher; reading them needs Viewer.

curl --fail-with-body \
  -H "Authorization: Bearer $JUDGMENT_API_KEY" \
  -H "X-Organization-Id: $JUDGMENT_ORG_ID" \
  -H "Content-Type: application/json" \
  -X POST "https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites/$TEST_SUITE_ID/runs" \
  --data '{"candidate": "'"$GIT_SHA"'", "trigger": "ci"}'