---
title: "Run a test config from your code"
deprecated: true
description: "Run your agent over a dataset from your own code and get each test case graded by the judges in a test config."
sidebar:
  label: "Run a test config from code"
seo:
  title: "Run a test config from your code | Judgment Docs"
  description: "Run your agent over a dataset from your own code and get each test case graded by the judges in a test config."
---

> **Test configs are deprecated**
>
> Test configs and dataset test runs (`client.offline_tests` / `client.offlineTests`)
> are available through the SDK and API only to organizations that used them
> before test suites launched. Other organizations get a `403` error. Use
> [test suites](/documentation/tests/overview) instead.

The SDK calls your agent once per test case, sends each trace to Judgment, waits while the judges grade them, and returns the results.

You need judgeval 1.3.3+ (Python, 3.10+) or judgeval 1.3.4+ (npm, Node 18+), a dataset, and a test config on it: the dataset plus the judges that grade it. In the app you set that up with **New config** on the dataset's page, or from code (below).

## How test config runs work

A test config pairs a dataset with the judges that grade it. Each run is one pass of your agent over one dataset version. Every run records the dataset version it used and the version of each judge that graded it, so you can tell which of them changed between two runs.

There are three ways to run a test config:

1. For an agent run, pass `agent_function`. The SDK runs your current agent on every test case and the judges grade the new traces. Use it when you change a prompt, a model, a tool, or code.
2. To re-score stored traces, leave out `agent_function`. The judges grade the trace each test case already points to. Use it when you change a judge and want to see how the new version labels the same traces. This needs a dataset with a trace field. You can also start a re-score from your coding agent with the MCP `run_test` tool, or by asking the Judgment Agent.
3. A CI gate is an agent run with a pass condition and `assert_test=True`. See [Run test configs in CI](/documentation/tests/run-tests-in-ci).

Agent runs produce one trace per test case, so choose trace-level judges for them. The app blocks judges that score whole sessions.

## The fast path: hand it to your coding agent

Start a new run on the dataset in the app. The **Run your agent** page has:

- **Options**: **Judge versions** (each judge uses its production version, or latest if none) and **Include pass condition (recommended)**.
- **Use your coding agent**: **Copy prompt**, or open it in Codex, Claude Code, Cursor or Devin. The prompt carries your project, dataset, version, judges and field names.
- **Or use the code template**: the same run as Python or TypeScript.

The page then shows "Waiting for an SDK run" and checks every few seconds. When your code sends the run, it switches to "New run received · View run".

## Write it yourself

Set `JUDGMENT_API_KEY` and `JUDGMENT_ORG_ID` in the environment. The client reads both. This example runs the [lunch-order agent](/documentation/start/example-agent) over the `lunch-regression` test config, whose judges are `request-completed` and `under-budget`.

**Python**

```python
import uuid

from judgeval import Judgeval
from lunch_agent import handle_request  # your agent's entry point

client = Judgeval(project_name="lunch-order-agent")
JUDGES = ("request-completed", "under-budget")

def lunch_agent(input: str) -> str:
    # One session per test case; handle_request takes (session_id, message).
    return handle_request(str(uuid.uuid4()), input)

def passed(fields, judge_results) -> bool:
    by_name = {r.name: r for r in judge_results}
    return all(
        name in by_name and by_name[name].error is None and by_name[name].value == "Yes"
        for name in JUDGES
    )

result = client.offline_tests.run(
    test_config="lunch-regression",
    agent_function=lunch_agent,
    pass_condition_fn=passed,
    run_name="lunch-agent-v2",
)
if result is None:
    raise RuntimeError("Project or test config not found")
print(result.status, result.passed, result.ui_results_url)
```

**TypeScript**

```typescript
import { Judgeval, type OfflineScorerData } from "judgeval";
import { randomUUID } from "node:crypto";
import { handleRequest } from "./lunch-agent"; // your agent's entry point

const client = await Judgeval.create({ projectName: "lunch-order-agent" });
const JUDGES = ["request-completed", "under-budget"];

const passed = (_fields: Record<string, unknown>, results: OfflineScorerData[]) =>
  JUDGES.every((name) => {
    const r = results.find((s) => s.name === name);
    return r !== undefined && r.error == null && r.value === "Yes";
  });

const result = await client.offlineTests.run("lunch-regression", {
  agentFunction: async (fields) => handleRequest(randomUUID(), String(fields.input)),
  passConditionFn: passed,
  runName: "lunch-agent-v2",
});
if (!result) throw new Error("Project or test config not found");
console.log(result.status, result.passed, result.uiResultsUrl);
```

The SDK wraps each call in an agent span, so model and tool calls you already trace with `wrap()` or `observe` show up as child spans in that test case's trace. Exceptions inside your agent are logged and the run moves on to the next case. Python runs cases one at a time; TypeScript takes `concurrency` (default 1).

## Map dataset fields to your agent

**Python** binds fields to parameters by name: `lunch_agent(input: str)` receives the `input` field. Fields the function doesn't declare are ignored, unless it takes `**kwargs`. When names differ, pass `field_mapping={"message": "input"}` so parameter `message` reads field `input`. **TypeScript** passes all fields as one object; read what you need, as in `fields.input` above. Only map the agent's real inputs; keep fields like `budget` or an expected answer for the pass condition.

## Write the pass condition

The pass condition gets the test case's fields and one result per judge, and returns `True` or `False`. Without one, the run records judge outputs only.

- A judge returns the label it chose, such as `"Yes"` or `"No"`. Check which one is good for each judge; "Yes" isn't always success.
- Older Score judges return a number.
- A result can carry an `error` or be missing. Decide what that means; the example above counts it as a failure.
- Judge reasoning and text values reach the SDK cut to 128 characters, so test values, not reasoning text.

## Pin versions

| Option (Python / TypeScript) | Default | Use it to |
|---|---|---|
| `dataset_version` / `datasetVersion` | Latest version | Run on the same test cases as an earlier run. Version number or version ID. |
| `judge_versions` / `judgeVersions` | Each judge's production version, else latest | Pin a judge: `[{"name": "under-budget", "version": "1.2"}]`. Also accepts `tag`, `judge_id`, or `major_version`/`minor_version`. Pin two versions of one judge to run both. |
| `run_name` / `runName` | Generated name | Label the run, for example with the agent version or commit. |
| `timeout_seconds` / `timeoutSeconds` | 600 | Wait longer for judge results. |
| `assert_test` / `assertTest` | `False` | Fail when any case fails. Needs a pass condition. See [Run test configs in CI](/documentation/tests/run-tests-in-ci). |

## Re-score stored traces

Leave out `agent_function` and the judges grade the trace each test case already points to. Use this after changing a judge: pin the new version with `judge_versions` and compare against the last run. The dataset needs a trace field.

## Create the test config from code

Python: `client.offline_tests.create_config(name="lunch-regression", dataset="lunch-regression", judges=["request-completed", "under-budget"])`. TypeScript: `client.offlineTests.createConfig(name, dataset, judges)`. `list_configs`, `get_config` and `delete_config` (camelCase in TypeScript) also exist. Judges must already exist; create them in the app, with the Judgment Agent, or with MCP `create_judge` (see [Create a judge](/documentation/judges/create-a-judge)).

## If something looks wrong

- If `run()` returns `None` / `null`, the project name or test config name wasn't found. The SDK logs which one.
- `TypeError: Agent entrypoint ... requires example field(s)` means a required parameter has no field with that name. Add `field_mapping`.
- "requires session scoring" shows up because agent runs make one trace per case, so session judges can't grade them. Use trace-level judges.
- A timeout (`TimeoutError` in Python, an `Error` in TypeScript) means the judges took longer than `timeout_seconds`. The run keeps going in Judgment; raise the timeout.
- If every Python case logs `asyncio.run() cannot be called from a running event loop`, call `run()` from synchronous code, not from inside an event loop. The SDK runs each async agent call with `asyncio.run`.
- If TypeScript throws "Offline tracer could not be activated", call `run()` outside any span you are already tracing.