---
title: "Create and edit a test suite"
description: "Write test cases with the judges to run and the answers you expect, then publish them as a version."
sidebar:
  label: "Create and edit a test suite"
seo:
  title: "Create and edit a test suite | Judgment Docs"
  description: "Create a test suite, add cases with expected judge answers, publish versions, and edit cases through a draft."
---

A test suite is a set of cases. Each case has an `input` for your agent and an
`expected` object that names the judges to run on the case's trace and the
result each judge should return. At the end of this guide, the test suite has a
published version that you can run.

**Prerequisites:** at least one [judge](/documentation/judges/overview) in the project,
with its ID and version (see [Find a judge's ID](#find-a-judges-id-and-version)).

## Create a test suite and add cases

```python
from judgeval import Judgeval

client = Judgeval(project_name="support-agent")
suite = client.test_suites.create(name="support-agent-regressions")

client.test_suites.add_cases(suite["test_suite_id"], [
    {
        "input": {"question": "I was charged twice"},
        "tags": ["billing"],
        "expected": {
            REQUEST_TYPE_JUDGE_ID: {  # a judge with the labels refund, billing, other
                "judge_major_version": 1, "judge_minor_version": 0,
                "op": "eq", "expected": "refund",
            },
            TOOL_CALLS_JUDGE_ID: {    # a judge that returns a number
                "judge_major_version": 1, "judge_minor_version": 0,
                "op": "lt", "expected": 3,
            },
        },
    },
])
client.test_suites.publish(suite["test_suite_id"])  # version 1 is now runnable
```

Each entry in `expected` is a **check**. It passes when the judge's output,
compared with `expected` using `op`, is true:

| `op` | Passes when |
| --- | --- |
| `eq`, `ne` | The output equals (or does not equal) `expected`. |
| `lt`, `lte`, `gt`, `gte` | The output compares to `expected` as a number. |
| `in`, `not_in` | The output is (or is not) in `expected`, a list. |

A judge's output is the label it chose, such as `"Yes"` or `"refund"`. Older
judges that return a score or true/false output that value instead.

Judges never receive the expected value, so the same judge can be used in
online monitoring and in test suites. A case runs only the judges listed in its
`expected`. If a judge ID or
version does not exist in the project, adding the case fails with a `422`. If a
judge is deleted after that, its check fails with `Judge not found` when the
case is scored.

Besides `input`, `expected` and `tags`, a case can have a `name` and a
`metadata` object. Only `input` is required. One call adds up to 1,000 cases.

## Find a judge's ID and version

- **In the app:** open the judge. Its ID is the last part of the page URL
  (`…/judges/<judge_id>`), and its versions are in the version menu on the
  judge page. `v1.2` is major version `1`, minor version `2`.
- **With MCP:** the `list_judges` and `get_judge` tools return a judge's ID and
  version. See [MCP tools](/documentation/reference/mcp-tools).

Use lowercase judge IDs as the keys of `expected`.

## Edit published cases

Published versions are immutable. Changes are made in the test suite's
**draft**, which becomes the next version when you publish it. New cases are
added directly to the draft. To edit or remove a published case, create the
draft first.

```python
client.test_suites.create_draft(test_suite_id)  # copy the published cases into the draft
client.test_suites.update_case(test_suite_id, case_id, {...})
client.test_suites.remove_case(test_suite_id, other_case_id)
client.test_suites.publish(test_suite_id)       # version 2
```

`client.test_suites.discard(test_suite_id)` discards the draft's changes. Earlier
versions, and the runs that used them, are unaffected.

`client.test_suites.delete(test_suite_id)` deletes the test suite with all its
versions, runs and results.

## Verify

`client.test_suites.list_cases(test_suite_id)` returns the latest published version's
cases. Pass `version="draft"` to see the draft, or a version number for an
older version.

In TypeScript, the same methods are on `judgeval.testSuites` in camelCase, such
as `addCases`, `createDraft` and `listCases`. See the
[Python](/sdk-reference/python/testing/TestSuitesFactory) and
[TypeScript](/sdk-reference/typescript/testing/test-suites-factory) SDK reference.

## Use the HTTP API

Every SDK call maps to a route under
`https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites`,
called with your API key and organization ID (see
[Authentication](/documentation/reference/authentication-and-projects)):

| SDK | Route |
| --- | --- |
| `test_suites.create(name)` | `POST /test-suites` |
| `test_suites.list()` | `GET /test-suites` |
| `test_suites.delete(test_suite_id)` | `DELETE /test-suites/{test_suite_id}` |
| `test_suites.list_cases(test_suite_id)` | `GET /test-suites/{test_suite_id}/cases` |
| `test_suites.add_cases(test_suite_id, cases)` | `POST /test-suites/{test_suite_id}/cases` |
| `test_suites.create_draft(test_suite_id)` | `POST /test-suites/{test_suite_id}/draft` |
| `test_suites.update_case(test_suite_id, case_id, case)` | `PUT /test-suites/{test_suite_id}/cases/{case_id}` |
| `test_suites.remove_case(test_suite_id, case_id)` | `DELETE /test-suites/{test_suite_id}/cases/{case_id}` |
| `test_suites.publish(test_suite_id)` | `POST /test-suites/{test_suite_id}/publish` |
| `test_suites.discard(test_suite_id)` | `POST /test-suites/{test_suite_id}/discard` |

Creating, editing, publishing and deleting need the Developer role or higher.

## Next step

[Run your agent on the test suite](/documentation/tests/run-a-test-suite).