Test your agent

Run your agent on a test suite's cases and check every run with your judges.

A test suite is a set of cases. Each case is an input for your agent, the judges to score it with, and the answer you expect from each judge. You send the input to your agent, and Judgment scores the trace it produces.

Use a test suite to:

• Gate pull requests in CI. Run every case and fail the build when a case fails.
• Compare models, prompts, or tools on the same cases.
• Test a deployed agent in staging or a preview deployment.
• Score a past run against changed cases without running the agent again.

Test suites are managed with the SDK or the HTTP API. The app has no pages for them.

How it works

Running a test suite takes two steps:

1. Start a run with the SDK or the API. You get back a run_id, and for each case its case_id and the input to send to your agent.

2. Run your agent once per case, and put two attributes on that case’s root span:

   Attribute	Value
   judgment.test.run_id	The run ID.
   judgment.test.case_id	The case ID.

Judgment scores each case as soon as its trace arrives. You can set the two attributes with any tracer, including plain OpenTelemetry that exports to Judgment:

with otel_tracer.start_as_current_span("support-agent") as span:  # your root span
    span.set_attribute("judgment.test.run_id", run_id)
    span.set_attribute("judgment.test.case_id", case_id)
    support_agent(case_input)

SDK helpers

The judgeval SDK provides optional helpers that reduce this setup:

• Tracer.test_case(run_id, case_id) wraps the agent call and tags the root span for you.
• headers: each started case includes a ready-made baggage header for an agent you call over HTTP.
• client.test_suites and client.test_runs in the SDK cover creating test suites, editing cases, starting runs, and reading results.

See Connect your agent for all supported ways to tag a trace.

How it fits together

A case passes when every judge listed in its expected returns the expected result. A judge error counts as a failed check.

Judge evaluations in a test run use Judgment Units (JU), like any other judge evaluation. See Billing and plans.

Next steps

• Run your first test suite end to end.
• Create and edit a test suite.
• Run your agent on a test suite in CI.

Test configs (legacy)

A test config pairs a dataset with the judges that grade it. The Datasets & Tests pages in the app still work with test configs, and the pages below cover that flow for organizations that already use it:

• Run a test config from your code
• Read and compare test config runs
• Run test configs in CI
• Cookbook: Testing with test configs