Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Run your first test suite

Create a test suite with one case, run a sample agent against it, and read the result.

In this tutorial, you create a judge and a test suite with one case, run a sample agent against it, and read the result. The tutorial takes a few minutes and uses one judge evaluation.

Prerequisites

  • The judgeval SDK, with the JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables set.
  • A project in Judgment. The script uses one named support-agent.

Create the judge

In the support-agent project, open Judges and create a judge that asks whether the final answer tells the customer they were refunded, with two labels: Yes and No. See Create a judge.

Open the judge and note two things for the script:

  • Its ID, the last part of the page URL (…/judges/<judge_id>).
  • Its version from the version menu. For v1.0, the major version is 1 and the minor version is 0.

Run the script

import time
from judgeval import Judgeval, Tracer

PROJECT = "support-agent"
JUDGE_ID = "<judge_id>"            # from the judge's page URL
JUDGE_MAJOR, JUDGE_MINOR = 1, 0    # from the judge's version menu

client = Judgeval(project_name=PROJECT)
Tracer.init(project_name=PROJECT)

# 1. A test suite with one case that expects the judge to return Yes.
suite = client.test_suites.create(name="my-first-test-suite")
client.test_suites.add_cases(suite["test_suite_id"], [{
    "input": {"question": "I was charged twice"},
    "expected": {
        JUDGE_ID: {
            "judge_major_version": JUDGE_MAJOR,
            "judge_minor_version": JUDGE_MINOR,
            "op": "eq",
            "expected": "Yes",
        },
    },
}])
client.test_suites.publish(suite["test_suite_id"])  # version 1 is now runnable

# 2. Start a run and run the agent once per case.
@Tracer.observe(span_type="agent")
def support_agent(question):
    return "Sorry about that! I've refunded the duplicate charge."

run = client.test_runs.start(suite["test_suite_id"])
for case in run["cases"]:
    with Tracer.test_case(run["run_id"], case["case_id"]):
        support_agent(case["input"]["question"])
Tracer.force_flush()

# 3. Wait until the case is scored, then print the result.
while client.test_runs.get(run["run_id"])["cases_scored"] < len(run["cases"]):
    time.sleep(5)
print(client.test_runs.results(run["run_id"]))

Test suite names must be unique in the project. To run the script again, change the name or delete the earlier test suite with client.test_suites.delete(test_suite_id).

Expected result

Within about a minute, the script prints a single passing case:

{
  "results": [{
    "case_id": "…", "trace_id": "…", "scored_at": "…", "passed": true,
    "checks": [{
      "judge_id": "…", "judge_major_version": 1, "judge_minor_version": 0,
      "op": "eq", "expected": "Yes", "judge_output": "Yes", "passed": true,
      "reason": "The final answer tells the customer they were refunded. <citation …>",
      "error_message": null
    }]
  }]
}

The reason cites the trace spans the judge relied on with <citation> tags, shortened here.

To see a failing case, change the sample agent’s answer so that it no longer mentions a refund, and start another run. The same case then fails with judge_output: "No".

Next steps

Was this page helpful?