---
title: "Turn production failures into regression tests"
description: "Carry one failed production trace into a dataset, a repeatable test, and a baseline-versus-candidate comparison."
sidebar:
  label: "Create regression cases"
seo:
  title: "Turn Production Failures into Regression Tests | Judgment Tutorial"
  description: "Follow a failed trace through dataset capture, an offline regression test, and a run comparison in Judgment."
---

This tutorial closes the production improvement loop with one concrete case:
a support agent answered without using the retrieved policy. You will preserve
the failed trace, add it to a reusable regression set, run a baseline and a
candidate, and compare the results.

## What you will produce

- One trace-backed dataset row with the failing evidence
- One reusable test config pairing that dataset with a judge
- Baseline and candidate runs pinned to known inputs
- One comparison that shows whether the failure changed

## Prerequisites

- A production trace with a verified failure
- A binary or categorical judge that represents the failure criterion
- A proposed agent or judge change to evaluate

If the failure is not localized yet, start with [Investigate a failed
trace](/documentation/monitoring/trace-investigation).

## 1. Preserve the failed trace

Open the failed trace and record the evidence that proves the problem: the
input, the span where execution diverged, the unsupported output, and the
relevant judge result or exception.

Select **Add to Dataset**. Create or choose a dataset named `Support
regressions`, then map:

- Trace input to `input`
- Trace output to `actual_output`
- Any declared context fields needed to reproduce the criterion

Judgment automatically fills the schema's `trace` field as a read-only link.
Confirm that the preview shows the source trace; do not map its ID manually.

If the dataset already declares `expected_output`, a category, or a
reviewer-note field, map a suitable trace value into it now. Every declared
field is required, and saving the first row locks the schema.

Select **Done** and open the dataset. Confirm that the latest version contains
the new row and that its trace field opens the original failure evidence.

## 2. Make the expectation explicit

Check that any mapped expected-output, category, or reviewer-note field says
what this case should do. If the trace does not contain a suitable value, keep
the expected criterion in the judge and use the trace plus its result as the
baseline evidence.

Do not add an undeclared field after the first row. Dataset schemas are locked
once examples exist; create a new dataset when the regression contract needs a
different shape.

## 3. Run the baseline

Open the dataset's **Tests** tab and select **Run Test**. Create a config such
as `Support policy regression` with the judge that checks whether the answer is
grounded in the retrieved policy.

In **Start a New Run**:

1. Name the run `Baseline — production failure`.
2. Select the dataset version containing the captured row.
3. Select the judge version that produced or represents the baseline decision.
4. Start the run and wait for the row's judge result.

Verify that the baseline result matches the failure you observed. If it does
not, calibrate the judge before using it as the regression criterion.

## 4. Run the candidate

Choose the path that matches the change:

- **Judge change:** start another platform run from the same config and dataset
  version, but select the candidate judge version.
- **Agent, model, prompt, or tool change:** [run the agent fresh through the
  SDK](/documentation/tests/offline-tests#run-your-agent-fresh-for-every-example)
  with the same config. The SDK calls the candidate once per example and
  attaches its fresh trace so the judge scores the new execution rather than
  the saved production trace.

Name the run for the candidate change. Keep the dataset version fixed so the
case roster does not move underneath the comparison. The current SDK run
options are documented in the [Python](/sdk-reference/python/offline_tests/offline_tests_factory)
and [TypeScript](/sdk-reference/typescript/offline_tests/offline-tests-factory)
references.

## 5. Compare the results

Open **Tests**, select the baseline and candidate runs, and choose **Compare**.
Filter to **Different Result** when the SDK reported pass conditions, or
**Different Judge Outputs** otherwise.

Open the captured row and compare:

- The baseline production trace and candidate trace, when the agent changed
- The selected judge versions, when the evaluator changed
- The judge values, reasons, and cited evidence
- The example fields that define the regression case

The tutorial succeeds when the comparison shows whether the candidate fixed
the known failure without changing the test input.

## Keep the loop running

After release, [monitor the behavior](/documentation/monitoring) on new
production traces. Add new failure shapes to the same dataset in a deliberate
batch, review the new dataset version, and run the regression config again.

This is the complete loop:

**failed production trace → dataset row → regression test → run comparison →
production monitoring**

## Troubleshooting

- If the baseline judge result is wrong, [calibrate the
  judge](/documentation/tests/judge-calibration) before comparing agent
  changes.
- If the candidate still shows the original trace, the run re-scored saved
  evidence. Use an SDK agent function for a fresh execution.
- If the comparison is unavailable, confirm that both runs use the same test
  config.
- If a newly added case is missing, verify that the run pinned the dataset
  version containing it.