Cookbook: Testing with test configs

Test a budget fix to the lunch-order agent with a coding agent, then gate pull requests on it.

Judgment gives you the infrastructure to test your agent before a change reaches production. You can rerun one test while you iterate on the agent, then use the same test as a CI gate on pull requests.

This cookbook does both with the lunch-order agent.

The change

A user asked the agent to “order vegetarian lunch for 12 under $200”. Version 1 searched nearby restaurants, compared menus, and recommended a $236 order. It finished the job, and it ignored the budget.

You want version 2 to respect budgets. The risk is the usual one with agents: you tighten the prompt, the agent gets cautious, and it stops finishing orders it used to finish. A unit test can’t catch that, because the output changes every run. What matters is the trajectory: did it check the price, and did it still place a sensible order?

Step 1: Write down the test cases

Start with ten real requests, including the ones that broke. Most teams pull these from production: the budget complaint came from a trace, so add that trace’s request.

from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="lunch-order-agent")

prompts = [
    "order vegetarian lunch for 12 under $200",
    "order mediterranean for 30",
    "I need halal catering for 20 by 11:30",
    "sandwiches for 8 people, keep it under $120",
    "something gluten free for the design team (15 people)",
    # ...five more from your traces
]

client.datasets.create(
    name="lunch-regression",
    schema={"type": "object", "properties": {"input": {"type": "string"}}},
    examples=[Example.create(input=p) for p in prompts],
)

You can also build the dataset in the app, including asking the agent to “Turn recent failures into test cases”. See Build a dataset.

Step 2: Decide what “good” means

Two judges grade every trace. Both are trace-level judges with two choices, Yes and No:

The judges read the whole trace, so under-budget sees the estimate_order call that returned $236, not only the final message. Create them in the app or ask the Judgment Agent to draft them; see Create a judge.

Then set up a test config on the dataset with those two judges. In code:

client.offline_tests.create_config(
    name="lunch-regression",
    dataset="lunch-regression",
    judges=["request-completed", "under-budget"],
)

Step 3: Get a baseline from v1

Put the run in a script you can call again and again. RUN_NAME labels each run so you can find it later, and dataset_version pins the test cases; use the number the dataset header shows (Version 1 here).

import os

import uuid

from judgeval import Judgeval
from lunch_agent import handle_request  # your agent's entry point

JUDGES = ("request-completed", "under-budget")
client = Judgeval(project_name="lunch-order-agent")

def lunch_agent(input: str) -> str:
    # One session per test case; handle_request takes (session_id, message).
    return handle_request(str(uuid.uuid4()), input)

def passed(fields, judge_results) -> bool:
    by_name = {r.name: r for r in judge_results}
    return all(
        name in by_name and by_name[name].error is None and by_name[name].value == "Yes"
        for name in JUDGES
    )

result = client.offline_tests.run(
    test_config="lunch-regression",
    agent_function=lunch_agent,
    pass_condition_fn=passed,
    dataset_version=1,
    run_name=os.environ.get("RUN_NAME", "local"),
)
if result is None:
    raise RuntimeError("Project or test config not found")

Run it on the current agent with RUN_NAME=lunch-agent-v1 python tests/run_lunch_test.py. The SDK calls the agent once per request, waits for the judges, and prints every case. A failing case looks like this:

• Example 1:
  request-completed: Yes (failed)
  under-budget: No (failed)

Pass or fail belongs to the whole case, so both judges show “failed” when one of them misses. The last line is a link to the run in Judgment. Say v1 passes 7 of 10: three requests with a budget came in over it.

Step 4: Change the agent and compare

Edit the system prompt so the agent compares the estimate_order total with the user’s budget before it recommends anything, and trims the order or tries the next restaurant when it’s over. Run the same script with RUN_NAME=lunch-agent-v2.

Now open the dataset in Judgment, select both runs, and choose Compare. Go straight to the Different Result tab. That’s the short list of requests where v2 behaves differently from v1:

• The three budget requests now pass under-budget.
• “I need halal catering for 20 by 11:30” now fails request-completed. Read the judge’s reasoning: it says v2 picked a cheaper restaurant whose check_delivery_eta came back after 11:30.

The fix made the agent cheaper, and along the way it stopped caring about the deadline. Both runs used dataset version 1 and the same judge versions, so the difference comes from your change. See Read and compare test config runs.

Step 5: Let a coding agent iterate

The loop above is: change something, run the script, read the failures, change something again. Coding agents are good at that loop, and the test gives them something better than “looks right to me”. Your coding agent does the editing; you review its diff.

Connect Judgment’s MCP server to Claude Code, Codex or Cursor first (Connect your coding agent), then give it a bounded task:

Run `RUN_NAME=v2-try-1 python tests/run_lunch_test.py`. It prints the judge
results for each test case and a link to the run. For each failing case, read
the judges' reasoning with the Judgment MCP tool get_test_run_items.

Change the system prompt in lunch_agent/prompts.py so the agent respects the
user's budget without missing their delivery time. Don't change the test, the
dataset, or the judges. Rerun with a new RUN_NAME (v2-try-2, v2-try-3) after
each change. Stop after three tries, then show me the diff and which cases
still fail.

Each attempt becomes its own run, so you can compare any two of them in the app. The same works when you have several coding agents try different approaches in parallel: give each one its own branch and its own RUN_NAME prefix, and compare the winners.

Two things keep the loop honest. Pin dataset_version so the test cases don’t change under the agent. And tell the agent not to touch the judges: a coding agent that can edit the rubric can make any test pass.

Step 6: Gate pull requests in CI

Once v2 passes, keep it passing: add assert_test=True to the run(...) call and run the script as a required check on pull requests that touch the agent. Run test configs in CI has the full job.

When the deadline regression shows up in production instead, add that trace’s request to the dataset and the gate covers it from then on. Adding cases creates a new dataset version, so bump dataset_version in the same pull request.

Track versions in production too

Run names label test runs. For production traces, set the judgment.agent_version attribute yourself (lunch-agent-v1, lunch-agent-v2) so you can split production results by version after you ship. See Add context.

What you did by hand

• Picked the test cases and kept them current.
• Wrote the judges and the pass condition.
• Started every run from your own code or CI. Judgment doesn’t start runs for you.
• Set up the coding-agent loop and reviewed its changes.