---
title: "Cookbook: Testing with test configs"
deprecated: true
description: "Test a budget fix to the lunch-order agent with a coding agent, then gate pull requests on it."
sidebar:
  label: "Cookbook: Test configs"
seo:
  title: "Cookbook: Testing with test configs | Judgment Docs"
  description: "Test a budget fix to the lunch-order agent with a coding agent, then gate pull requests on it."
---

> **Uses deprecated test configs**
>
> This cookbook uses test configs (`client.offline_tests`), which the SDK and
> API allow only for organizations that used them before test suites launched.
> New projects should use [test suites](/documentation/tests/overview); start
> with [Run your first test suite](/documentation/tests/first-test-suite).

Judgment gives you the infrastructure to test your agent before a change reaches production. You can rerun one test while you iterate on the agent, then use the same test as a CI gate on pull requests.

This cookbook does both with the [lunch-order agent](/documentation/start/example-agent).

## The change

A user asked the agent to "order vegetarian lunch for 12 under $200". Version 1 searched nearby restaurants, compared menus, and recommended a $236 order. It finished the job, and it ignored the budget.

You want version 2 to respect budgets. The risk is the usual one with agents: you tighten the prompt, the agent gets cautious, and it stops finishing orders it used to finish. A unit test can't catch that, because the output changes every run. What matters is the trajectory: did it check the price, and did it still place a sensible order?

## Step 1: Write down the test cases

Start with ten real requests, including the ones that broke. Most teams pull these from production: the budget complaint came from a trace, so add that trace's request.

```python
from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="lunch-order-agent")

prompts = [
    "order vegetarian lunch for 12 under $200",
    "order mediterranean for 30",
    "I need halal catering for 20 by 11:30",
    "sandwiches for 8 people, keep it under $120",
    "something gluten free for the design team (15 people)",
    # ...five more from your traces
]

client.datasets.create(
    name="lunch-regression",
    schema={"type": "object", "properties": {"input": {"type": "string"}}},
    examples=[Example.create(input=p) for p in prompts],
)
```

You can also build the dataset in the app, including asking the agent to "Turn recent failures into test cases". See [Build a dataset](/documentation/tests/build-a-dataset).

## Step 2: Decide what "good" means

Two judges grade every trace. Both are trace-level judges with two choices, Yes and No:

| Judge | Question it answers | Good answer |
|---|---|---|
| `request-completed` | Did the agent end with an order recommendation that matches the request (cuisine, headcount, timing)? | Yes |
| `under-budget` | If the user stated a budget, does the recommended total stay under it? Answer Yes when no budget was stated. | Yes |

The judges read the whole trace, so `under-budget` sees the `estimate_order` call that returned $236, not only the final message. Create them in the app or ask the Judgment Agent to draft them; see [Create a judge](/documentation/judges/create-a-judge).

Then set up a test config on the dataset with those two judges. In code:

```python
client.offline_tests.create_config(
    name="lunch-regression",
    dataset="lunch-regression",
    judges=["request-completed", "under-budget"],
)
```

## Step 3: Get a baseline from v1

Put the run in a script you can call again and again. `RUN_NAME` labels each run so you can find it later, and `dataset_version` pins the test cases; use the number the dataset header shows (**Version 1** here).

```python tests/run_lunch_test.py
import os

import uuid

from judgeval import Judgeval
from lunch_agent import handle_request  # your agent's entry point

JUDGES = ("request-completed", "under-budget")
client = Judgeval(project_name="lunch-order-agent")

def lunch_agent(input: str) -> str:
    # One session per test case; handle_request takes (session_id, message).
    return handle_request(str(uuid.uuid4()), input)

def passed(fields, judge_results) -> bool:
    by_name = {r.name: r for r in judge_results}
    return all(
        name in by_name and by_name[name].error is None and by_name[name].value == "Yes"
        for name in JUDGES
    )

result = client.offline_tests.run(
    test_config="lunch-regression",
    agent_function=lunch_agent,
    pass_condition_fn=passed,
    dataset_version=1,
    run_name=os.environ.get("RUN_NAME", "local"),
)
if result is None:
    raise RuntimeError("Project or test config not found")
```

Run it on the current agent with `RUN_NAME=lunch-agent-v1 python tests/run_lunch_test.py`. The SDK calls the agent once per request, waits for the judges, and prints every case. A failing case looks like this:

```text
• Example 1:
  request-completed: Yes (failed)
  under-budget: No (failed)
```

Pass or fail belongs to the whole case, so both judges show "failed" when one of them misses. The last line is a link to the run in Judgment. Say v1 passes 7 of 10: three requests with a budget came in over it.

## Step 4: Change the agent and compare

Edit the system prompt so the agent compares the `estimate_order` total with the user's budget before it recommends anything, and trims the order or tries the next restaurant when it's over. Run the same script with `RUN_NAME=lunch-agent-v2`.

Now open the dataset in Judgment, select both runs, and choose **Compare**. Go straight to the **Different Result** tab. That's the short list of requests where v2 behaves differently from v1:

- The three budget requests now pass `under-budget`.
- "I need halal catering for 20 by 11:30" now fails `request-completed`. Read the judge's reasoning: it says v2 picked a cheaper restaurant whose `check_delivery_eta` came back after 11:30.

The fix made the agent cheaper, and along the way it stopped caring about the deadline. Both runs used dataset version 1 and the same judge versions, so the difference comes from your change. See [Read and compare test config runs](/documentation/tests/compare-runs).

## Step 5: Let a coding agent iterate

The loop above is: change something, run the script, read the failures, change something again. Coding agents are good at that loop, and the test gives them something better than "looks right to me". Your coding agent does the editing; you review its diff.

Connect Judgment's MCP server to Claude Code, Codex or Cursor first ([Connect your coding agent](/documentation/connect/connect-your-coding-agent)), then give it a bounded task:

```text
Run `RUN_NAME=v2-try-1 python tests/run_lunch_test.py`. It prints the judge
results for each test case and a link to the run. For each failing case, read
the judges' reasoning with the Judgment MCP tool get_test_run_items.

Change the system prompt in lunch_agent/prompts.py so the agent respects the
user's budget without missing their delivery time. Don't change the test, the
dataset, or the judges. Rerun with a new RUN_NAME (v2-try-2, v2-try-3) after
each change. Stop after three tries, then show me the diff and which cases
still fail.
```

Each attempt becomes its own run, so you can compare any two of them in the app. The same works when you have several coding agents try different approaches in parallel: give each one its own branch and its own `RUN_NAME` prefix, and compare the winners.

Two things keep the loop honest. Pin `dataset_version` so the test cases don't change under the agent. And tell the agent not to touch the judges: a coding agent that can edit the rubric can make any test pass.

## Step 6: Gate pull requests in CI

Once v2 passes, keep it passing: add `assert_test=True` to the `run(...)` call and run the script as a required check on pull requests that touch the agent. [Run test configs in CI](/documentation/tests/run-tests-in-ci) has the full job.

When the deadline regression shows up in production instead, add that trace's request to the dataset and the gate covers it from then on. Adding cases creates a new dataset version, so bump `dataset_version` in the same pull request.

## Track versions in production too

Run names label test runs. For production traces, set the `judgment.agent_version` attribute yourself (`lunch-agent-v1`, `lunch-agent-v2`) so you can split production results by version after you ship. See [Add context](/documentation/connect/add-context).

## What you did by hand

- Picked the test cases and kept them current.
- Wrote the judges and the pass condition.
- Started every run from your own code or CI. Judgment doesn't start runs for you.
- Set up the coding-agent loop and reviewed its changes.