Run an offline test

Evaluate one dataset version with selected judges and inspect the pinned results in Judgment.

An offline test evaluates a fixed dataset with reusable judges, independently of live monitoring. At the end of this guide, one run has completed and its dataset and judge versions are visible in Judgment.

Use an offline test to answer one change question against evidence that stays fixed. A typical regression workflow is:

Use saved traces when you are evaluating a judge change. Run the agent through the SDK when you are evaluating an agent, model, prompt, or tool change, so the candidate produces fresh trace evidence for every dataset example.

Prerequisites

• A dataset with at least one example
• A trace or session field when starting the run from the platform
• At least one compatible judge with a saved version

For an input-only dataset, use an SDK run with an agent function so each example produces a fresh trace before judging.

Run the test

Expected result

The run reaches Completed, every dataset row has a result for each resolved judge version, and the run list shows:

• The config and run name
• The pinned dataset version
• Each judge and exact major/minor version
• The source and completion status

Pass conditions

Judge outputs are neutral evaluation results. A platform run does not invent a pass/fail rule. SDK runs show a passed ratio only when they define a per-example pass condition.

Run your agent fresh for every example

Platform-started runs score the trace or session already stored in each row. To evaluate an agent, prompt, model, or tool change against literal dataset inputs, use the SDK’s optional agent function. The SDK calls it once per example, attaches the new trace to the run, and can record a per-row pass condition.

The examples below continue the regression-case tutorial. They use its Support policy regression config and required input string field. Replace dataset version 2 with the exact version used by your baseline, and replace the candidate function body with your agent call.

from judgeval import Judgeval

client = Judgeval(project_name="default_project")

def candidate_agent(input: str) -> str:
    # Replace this body with the candidate agent, prompt, model, or tools.
    return f"Candidate response to: {input}"

def passed(fields, scorers) -> bool:
    return all(s.error is None and s.value == "Yes" for s in scorers)

result = client.offline_tests.run(
    test_config="Support policy regression",
    agent_function=candidate_agent,
    dataset_version=2,
    pass_condition_fn=passed,
    run_name="Candidate — agent change",
)
if result is None:
    raise RuntimeError("Offline test could not be created")

print(result.ui_results_url)

import { Judgeval } from "judgeval";

const client = await Judgeval.create({ projectName: "default_project" });

const candidateAgent = async (
  fields: Record<string, unknown>,
): Promise<string> => {
  const input = String(fields.input);
  // Replace this body with the candidate agent, prompt, model, or tools.
  return `Candidate response to: ${input}`;
};

const result = await client.offlineTests.run(
  "Support policy regression",
  {
    agentFunction: candidateAgent,
    datasetVersion: 2,
    passConditionFn: (_fields, scorers) =>
      scorers.every((s) => s.error == null && s.value === "Yes"),
    runName: "Candidate — agent change",
  },
);
if (!result) throw new Error("Offline test could not be created");

console.log(result.uiResultsUrl);

The runner activates an offline tracer and wraps the candidate in an agent span, so do not initialize another top-level tracer for this call. Traced model, tool, and retrieval work inside your candidate becomes child-span evidence. The runner then creates the test run with one fresh trace per example and waits for the judge results. The pass callback stores one pass/fail outcome per row; this example passes only when every judge returns a successful binary Yes.

See the Python offline tests reference or the TypeScript offline tests reference for the current run signatures and options.

Troubleshooting

• If the dataset does not appear, it may be a read-only legacy dataset.
• If the config cannot be created or run, add one trace or session field to the dataset, or use an SDK agent function to supply a trace for every row.
• If a session-level judge is rejected, use a session-backed dataset. A trace-only dataset does not provide the whole session it expects.
• If Start Run is disabled, choose a dataset version that contains at least one example.
• If a judge returns an error, inspect that result separately from the agent output. An evaluation failure is not an agent failure.
• If the results do not define success, configure a pass condition in the SDK or compare the judge values directly.

Next step

Compare test runs after you have a baseline and a candidate run from the same config.