---
title: "Run your agent on a test suite"
description: "Start a run, run your agent once per case, and read each case's result."
sidebar:
  label: "Run your agent on a test suite"
seo:
  title: "Run your agent on a test suite | Judgment Docs"
  description: "Start a test run with the SDK or API, run your agent once per case, read results, gate CI, and score an existing run."
---

A run has four steps: start the run, run your agent once per case, wait for the
cases to be scored, and read the results. Judgment does not call your agent;
your code runs it.

**Prerequisites:** a [test suite](/documentation/tests/create-a-test-suite) with at
least one published version, and your API key, organization ID, and project
name (see
[Authentication](/documentation/reference/authentication-and-projects)).

## Start a run and run your agent

Start a run, then run your agent once per case with `judgment.test.run_id` and
`judgment.test.case_id` on the case's root span. The examples use the
`Tracer.test_case()` wrapper; setting the two attributes directly is
equivalent (see [Connect your agent](/documentation/tests/connect-your-agent)).

**Python**

```python
import os

from judgeval import Judgeval, Tracer

client = Judgeval(project_name="support-agent")
run = client.test_runs.start(
    TEST_SUITE_ID, candidate=os.environ["GIT_SHA"], trigger="ci"
)

for case in run["cases"]:
    with Tracer.test_case(run["run_id"], case["case_id"]):
        support_agent(case["input"])
```

**TypeScript**

```typescript
import { Judgeval, Tracer } from "judgeval";

const judgeval = await Judgeval.create({ projectName: "support-agent" });
const run = await judgeval.testRuns.start(TEST_SUITE_ID, {
    candidate: process.env.GIT_SHA,
    trigger: "ci",
});

for (const c of run.cases) {
    await Tracer.testCase(run.run_id, c.case_id, () => supportAgent(c.input));
}
```

The run uses the test suite's latest published version. Each case in `run["cases"]`
has its `case_id`, the `input` to send to your agent, and the tagging `headers`
for an agent you call over HTTP. Judgment scores each case as soon as its trace
arrives.

`start` accepts the following optional parameters:

| Option | Meaning |
| --- | --- |
| `case_filter` | Which cases to run: `{"kind": "all"}` (default), `{"kind": "tags", "tags": [...]}`, `{"kind": "case_ids", "case_ids": [...]}`, or `{"kind": "last_failures", "run_id": "..."}`. |
| `candidate` | What is being tested, such as a git SHA or agent version. |
| `candidate_meta` | Any other details about the candidate, as an object. |
| `trigger` | `manual`, `ci`, `schedule`, or `api` (default). |

## Read results

`client.test_runs.get(run_id)` returns the run's counts:

| Field | Meaning |
| --- | --- |
| `cases_total` | Cases in the run. |
| `cases_with_output` | Cases whose trace has arrived. |
| `cases_scored` | Cases that have been scored. |
| `cases_passed` | Scored cases whose checks all passed. |

The run is finished when `cases_scored` equals `cases_total`.
`client.test_runs.list(test_suite_id)` lists a test suite's runs, newest first.

`client.test_runs.results(run_id)` returns one entry per case, with one entry
per check:

```json
{
  "case_id": "…", "trace_id": "…", "scored_at": "…", "passed": false,
  "checks": [{
    "judge_id": "…", "judge_major_version": 1, "judge_minor_version": 0,
    "op": "eq", "expected": "Yes", "judge_output": "No", "passed": false,
    "reason": "The final answer does not mention a refund. <citation …>",
    "error_message": null
  }]
}
```

## Fail a CI job on a failed case

To use a run as a CI gate, wait until every case is scored and exit with an
error when any case did not pass:

```python
import sys
import time

Tracer.force_flush()
while (summary := client.test_runs.get(run["run_id"]))["cases_scored"] < summary["cases_total"]:
    time.sleep(5)
if summary["cases_passed"] < summary["cases_total"]:
    sys.exit(f"{summary['cases_total'] - summary['cases_passed']} test cases failed")
```

A run has no deadline, so give the CI job a timeout in case a trace never
arrives (see [a case is never scored](#troubleshooting-a-case-is-never-scored)).
Store `JUDGMENT_API_KEY` and `JUDGMENT_ORG_ID` as CI secrets.

## Retry failures

Start a new run with `{"kind": "last_failures", "run_id": "<earlier run>"}` as
its `case_filter`. It runs only the cases that failed or were never scored in
that run.

## Score an existing run

After you change a test suite's cases and publish a new version, you can score
the existing traces again without rerunning your agent:

```python
new_run = client.test_runs.score(run_id)
```

This creates a new run, scored with the test suite's latest published version, from
the cases that have a trace and whose `input` did not change. The original run
is not modified. The new run's cases are scored in the background, so poll
`client.test_runs.get(new_run["run_id"])` for progress. When the run already
used the latest version, scoring it again retries cases that were never scored.

## Troubleshooting: a case is never scored

A run has no deadline, so a case waits until its trace arrives. If
`cases_with_output` stays below `cases_total`, check that:

- **The attributes are on the root span**, not a child span.
- **The trace lands in the test suite's project.**
- **Only `baggage` is sent** to a deployed agent. A `traceparent` from a span
  Judgment never receives leaves the case without a root span.
- **Each case runs as its own trace.** Judgment scores the first trace for each
  case and ignores later traces.

If a check fails with `error_message: "Judge not found"`, its judge was
deleted after the case was created. A check can also fail with
`rate limited` or `usage limit reached` when your organization hits a limit.

## Use the HTTP API

Every SDK call maps to a route under
`https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites`, called
with your API key and organization ID:

| SDK | Route |
| --- | --- |
| `test_runs.start(test_suite_id)` | `POST /test-suites/{test_suite_id}/runs` |
| `test_runs.list(test_suite_id)` | `GET /test-suites/{test_suite_id}/runs` |
| `test_runs.get(run_id)` | `GET /test-suites/runs/{run_id}` |
| `test_runs.results(run_id)` | `GET /test-suites/runs/{run_id}/results` |
| `test_runs.score(run_id)` | `POST /test-suites/runs/{run_id}/score` |

Starting and scoring runs need the Developer role or higher; reading them needs
Viewer.

```bash
curl --fail-with-body \
  -H "Authorization: Bearer $JUDGMENT_API_KEY" \
  -H "X-Organization-Id: $JUDGMENT_ORG_ID" \
  -H "Content-Type: application/json" \
  -X POST "https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites/$TEST_SUITE_ID/runs" \
  --data '{"candidate": "'"$GIT_SHA"'", "trigger": "ci"}'
```