---
title: "Test your agent"
description: "Run your agent on a test suite's cases and check every run with your judges."
sidebar:
  label: "Overview"
seo:
  title: "Test your agent | Judgment Docs"
  description: "Start a test run, tag each case's root span with two attributes, and Judgment checks every case with your judges."
---

A **test suite** is a set of **cases**. Each case is an input for your agent,
the judges to score it with, and the answer you expect from each judge. You
send the input to your agent, and Judgment scores the trace it produces.

Use a test suite to:

- **Gate pull requests in CI.** Run every case and fail the build when a case
  fails.
- **Compare models, prompts, or tools** on the same cases.
- **Test a deployed agent** in staging or a preview deployment.
- **Score a past run against changed cases** without running the agent again.

Test suites are managed with the SDK or the HTTP API. The app has no pages for
them.

## How it works

Running a test suite takes two steps:

1. **Start a run** with the SDK or the API. You get back a `run_id`, and for each
   case its `case_id` and the `input` to send to your agent.
2. **Run your agent once per case**, and put two attributes on that case's
   **root span**:

   | Attribute | Value |
   | --- | --- |
   | `judgment.test.run_id` | The run ID. |
   | `judgment.test.case_id` | The case ID. |

Judgment scores each case as soon as its trace arrives. You can set the two
attributes with any tracer, including plain OpenTelemetry that
[exports to Judgment](/documentation/connect/integrations/send-otlp-directly):

```python
with otel_tracer.start_as_current_span("support-agent") as span:  # your root span
    span.set_attribute("judgment.test.run_id", run_id)
    span.set_attribute("judgment.test.case_id", case_id)
    support_agent(case_input)
```

### SDK helpers

The judgeval SDK provides optional helpers that reduce this setup:

- **`Tracer.test_case(run_id, case_id)`** wraps the agent call and tags the root
  span for you.
- **`headers`**: each started case includes a ready-made `baggage` header for an
  agent you call over HTTP.
- **`client.test_suites` and `client.test_runs`** in the SDK cover creating test
  suites, editing cases, starting runs, and reading results.

See [Connect your agent](/documentation/tests/connect-your-agent) for all
supported ways to tag a trace.

## How it fits together

| Part | What it is |
| --- | --- |
| **Test suite** | A named set of cases, published as numbered versions. Runs use a published version. |
| **Case** | The `input` your agent gets, plus `expected`: which [judges](/documentation/judges/overview) to run on its trace and the result each should return. |
| **Run** | One pass over the test suite's cases. Each case produces exactly one trace. |

A case **passes** when every judge listed in its `expected` returns the
expected result. A judge error counts as a failed check.

Judge evaluations in a test run use Judgment Units (JU), like any other judge
evaluation. See [Billing and plans](/documentation/administration/billing-and-plans).

## Next steps

- [Run your first test suite](/documentation/tests/first-test-suite) end to end.
- [Create and edit a test suite](/documentation/tests/create-a-test-suite).
- [Run your agent on a test suite](/documentation/tests/run-a-test-suite) in CI.

## Test configs (legacy)

> **Info**
>
> Test configs and dataset test runs (`client.offline_tests` in Python,
> `client.offlineTests` in TypeScript) are deprecated. Through the SDK and API
> they are available only to organizations that used them before test suites
> launched; other organizations get a `403` error that links to this page. Use
> test suites instead. Datasets themselves are not deprecated.

A **test config** pairs a dataset with the judges that grade it. The
**Datasets & Tests** pages in the app still work with test configs, and the
pages below cover that flow for organizations that already use it:

- [Run a test config from your code](/documentation/tests/run-a-test-from-your-code)
- [Read and compare test config runs](/documentation/tests/compare-runs)
- [Run test configs in CI](/documentation/tests/run-tests-in-ci)
- [Cookbook: Testing with test configs](/documentation/tests/cookbook-testing)