---
title: "Run an offline test"
description: "Evaluate one dataset version with selected judges and inspect the pinned results in Judgment."
sidebar:
  label: "Run an offline test"
seo:
  title: "Run an Offline Test | Judgment How-to"
  description: "Pair a dataset with judges, select dataset and judge versions, start an offline run, and inspect its results."
---

An offline test evaluates a fixed dataset with reusable judges, independently
of live monitoring. At the end of this guide, one run has completed and its
dataset and judge versions are visible in Judgment.

Use an offline test to answer one change question against evidence that stays
fixed. A typical regression workflow is:

```mermaid
flowchart TD
  Trace["Production trace"] -->|"saved in"| Dataset["Dataset version"]
  Dataset -->|"used by"| Baseline["Baseline run"]
  Dataset -->|"used by"| Candidate["Candidate run"]
  Baseline -->|"compared with"| Comparison["Comparison"]
  Candidate -->|"compared with"| Comparison
```

Use saved traces when you are evaluating a judge change. Run the agent through
the SDK when you are evaluating an agent, model, prompt, or tool change, so the
candidate produces fresh trace evidence for every dataset example.

## Prerequisites

- A dataset with at least one example
- A `trace` or `session` field when starting the run from the platform
- At least one compatible judge with a saved version

For an input-only dataset, use an SDK run with an agent function so each
example produces a fresh trace before judging.

## Run the test

1. ### Start from the dataset

    Open **Datasets**, choose the dataset, and select its **Tests** tab. Select
    **Run Test**.

    Starting here keeps the input asset visible and limits the config picker to
    tests that use this dataset. You can also open **Tests** and choose **New
    Config**.

2. ### Choose or create the config

    Select an existing config when it already represents the dataset and judge
    set you need. Otherwise select **Create new test config**.

    Give the config a durable name and choose one or more judges. A config is a
    reusable binding between one dataset and a judge set; it is not a run and
    does not pin versions by itself.

    Leave **Run after creating** enabled, then select **Create & Run**.

3. ### Pin the run inputs

    In **Start a New Run**, name the run for the change or baseline it
    represents. Choose the **Dataset Version** you intend to evaluate. The
    latest version is the default.

    Review **Judge versions** and explicitly select versions when the run must
    be reproducible or compare an evaluator change. When no override is sent,
    the service resolves each judge to its `prod` version when one exists,
    otherwise its latest version.

    Judgment resolves the dataset and judges to exact versions when the run is
    prepared. Later edits do not change an existing run.

4. ### Start and inspect the run

    Select **Start Run**. Open the run from the dataset's Tests tab or from
    **Tests**.

    The run moves through pending and running states as each row is scored.
    Inspect the judge breakdown, then open individual rows to read their data,
    saved trace or session, judge values, reasons, and errors.

## Expected result

The run reaches **Completed**, every dataset row has a result for each resolved
judge version, and the run list shows:

- The config and run name
- The pinned dataset version
- Each judge and exact major/minor version
- The source and completion status

## Pass conditions

Judge outputs are neutral evaluation results. A platform run does not invent a
pass/fail rule. SDK runs show a passed ratio only when they define a per-example
pass condition.

## Run your agent fresh for every example

Platform-started runs score the trace or session already stored in each row.
To evaluate an agent, prompt, model, or tool change against literal dataset
inputs, use the SDK's optional agent function. The SDK calls it once per
example, attaches the new trace to the run, and can record a per-row pass
condition.

The examples below continue the [regression-case
tutorial](/documentation/tests/regression-cases). They use its `Support policy
regression` config and required `input` string field. Replace dataset version
`2` with the exact version used by your baseline, and replace the candidate
function body with your agent call.

**Python**

```python title="offline_test.py"
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

def candidate_agent(input: str) -> str:
    # Replace this body with the candidate agent, prompt, model, or tools.
    return f"Candidate response to: {input}"

def passed(fields, scorers) -> bool:
    return all(s.error is None and s.value == "Yes" for s in scorers)

result = client.offline_tests.run(
    test_config="Support policy regression",
    agent_function=candidate_agent,
    dataset_version=2,
    pass_condition_fn=passed,
    run_name="Candidate — agent change",
)
if result is None:
    raise RuntimeError("Offline test could not be created")

print(result.ui_results_url)
```

**TypeScript**

```typescript title="offlineTest.ts"
import { Judgeval } from "judgeval";

const client = await Judgeval.create({ projectName: "default_project" });

const candidateAgent = async (
  fields: Record<string, unknown>,
): Promise<string> => {
  const input = String(fields.input);
  // Replace this body with the candidate agent, prompt, model, or tools.
  return `Candidate response to: ${input}`;
};

const result = await client.offlineTests.run(
  "Support policy regression",
  {
    agentFunction: candidateAgent,
    datasetVersion: 2,
    passConditionFn: (_fields, scorers) =>
      scorers.every((s) => s.error == null && s.value === "Yes"),
    runName: "Candidate — agent change",
  },
);
if (!result) throw new Error("Offline test could not be created");

console.log(result.uiResultsUrl);
```

The runner activates an offline tracer and wraps the candidate in an agent
span, so do not initialize another top-level tracer for this call. Traced model,
tool, and retrieval work inside your candidate becomes child-span evidence.
The runner then creates the test run with one fresh trace per example and waits
for the judge results. The pass callback stores one pass/fail outcome per row;
this example passes only when every judge returns a successful binary `Yes`.

See the [Python offline tests
reference](/sdk-reference/python/offline_tests/offline_tests_factory) or the
[TypeScript offline tests
reference](/sdk-reference/typescript/offline_tests/offline-tests-factory) for
the current run signatures and options.

## Troubleshooting

- If the dataset does not appear, it may be a read-only legacy dataset.
- If the config cannot be created or run, add one `trace` or `session` field to
  the dataset, or use an SDK agent function to supply a trace for every row.
- If a session-level judge is rejected, use a session-backed dataset. A
  trace-only dataset does not provide the whole session it expects.
- If **Start Run** is disabled, choose a dataset version that contains at least
  one example.
- If a judge returns an error, inspect that result separately from the agent
  output. An evaluation failure is not an agent failure.
- If the results do not define success, configure a pass condition in the SDK
  or compare the judge values directly.

## Next step

[Compare test runs](/documentation/tests/test-run-comparison) after you have a
baseline and a candidate run from the same config.