Create and edit a test suite

Write test cases with the judges to run and the answers you expect, then publish them as a version.

A test suite is a set of cases. Each case has an input for your agent and an expected object that names the judges to run on the case’s trace and the result each judge should return. At the end of this guide, the test suite has a published version that you can run.

Prerequisites: at least one judge in the project, with its ID and version (see Find a judge’s ID).

Create a test suite and add cases

from judgeval import Judgeval

client = Judgeval(project_name="support-agent")
suite = client.test_suites.create(name="support-agent-regressions")

client.test_suites.add_cases(suite["test_suite_id"], [
    {
        "input": {"question": "I was charged twice"},
        "tags": ["billing"],
        "expected": {
            REQUEST_TYPE_JUDGE_ID: {  # a judge with the labels refund, billing, other
                "judge_major_version": 1, "judge_minor_version": 0,
                "op": "eq", "expected": "refund",
            },
            TOOL_CALLS_JUDGE_ID: {    # a judge that returns a number
                "judge_major_version": 1, "judge_minor_version": 0,
                "op": "lt", "expected": 3,
            },
        },
    },
])
client.test_suites.publish(suite["test_suite_id"])  # version 1 is now runnable

Each entry in expected is a check. It passes when the judge’s output, compared with expected using op, is true:

A judge’s output is the label it chose, such as "Yes" or "refund". Older judges that return a score or true/false output that value instead.

Judges never receive the expected value, so the same judge can be used in online monitoring and in test suites. A case runs only the judges listed in its expected. If a judge ID or version does not exist in the project, adding the case fails with a 422. If a judge is deleted after that, its check fails with Judge not found when the case is scored.

Besides input, expected and tags, a case can have a name and a metadata object. Only input is required. One call adds up to 1,000 cases.

Find a judge’s ID and version

• In the app: open the judge. Its ID is the last part of the page URL (…/judges/<judge_id>), and its versions are in the version menu on the judge page. v1.2 is major version 1, minor version 2.
• With MCP: the list_judges and get_judge tools return a judge’s ID and version. See MCP tools.

Use lowercase judge IDs as the keys of expected.

Edit published cases

Published versions are immutable. Changes are made in the test suite’s draft, which becomes the next version when you publish it. New cases are added directly to the draft. To edit or remove a published case, create the draft first.

client.test_suites.create_draft(test_suite_id)  # copy the published cases into the draft
client.test_suites.update_case(test_suite_id, case_id, {...})
client.test_suites.remove_case(test_suite_id, other_case_id)
client.test_suites.publish(test_suite_id)       # version 2

client.test_suites.discard(test_suite_id) discards the draft’s changes. Earlier versions, and the runs that used them, are unaffected.

client.test_suites.delete(test_suite_id) deletes the test suite with all its versions, runs and results.

Verify

client.test_suites.list_cases(test_suite_id) returns the latest published version’s cases. Pass version="draft" to see the draft, or a version number for an older version.

In TypeScript, the same methods are on judgeval.testSuites in camelCase, such as addCases, createDraft and listCases. See the Python and TypeScript SDK reference.

Use the HTTP API

Every SDK call maps to a route under https://api.judgmentlabs.ai/v1/projects/$JUDGMENT_PROJECT_ID/test-suites, called with your API key and organization ID (see Authentication):

Creating, editing, publishing and deleting need the Developer role or higher.

Next step

Run your agent on the test suite.