Turn production failures into regression tests

Carry one failed production trace into a dataset, a repeatable test, and a baseline-versus-candidate comparison.

This tutorial closes the production improvement loop with one concrete case: a support agent answered without using the retrieved policy. You will preserve the failed trace, add it to a reusable regression set, run a baseline and a candidate, and compare the results.

What you will produce

• One trace-backed dataset row with the failing evidence
• One reusable test config pairing that dataset with a judge
• Baseline and candidate runs pinned to known inputs
• One comparison that shows whether the failure changed

Prerequisites

• A production trace with a verified failure
• A binary or categorical judge that represents the failure criterion
• A proposed agent or judge change to evaluate

If the failure is not localized yet, start with Investigate a failed trace.

1. Preserve the failed trace

Open the failed trace and record the evidence that proves the problem: the input, the span where execution diverged, the unsupported output, and the relevant judge result or exception.

Select Add to Dataset. Create or choose a dataset named Support regressions, then map:

• Trace input to input
• Trace output to actual_output
• Any declared context fields needed to reproduce the criterion

Judgment automatically fills the schema’s trace field as a read-only link. Confirm that the preview shows the source trace; do not map its ID manually.

If the dataset already declares expected_output, a category, or a reviewer-note field, map a suitable trace value into it now. Every declared field is required, and saving the first row locks the schema.

Select Done and open the dataset. Confirm that the latest version contains the new row and that its trace field opens the original failure evidence.

2. Make the expectation explicit

Check that any mapped expected-output, category, or reviewer-note field says what this case should do. If the trace does not contain a suitable value, keep the expected criterion in the judge and use the trace plus its result as the baseline evidence.

Do not add an undeclared field after the first row. Dataset schemas are locked once examples exist; create a new dataset when the regression contract needs a different shape.

3. Run the baseline

Open the dataset’s Tests tab and select Run Test. Create a config such as Support policy regression with the judge that checks whether the answer is grounded in the retrieved policy.

In Start a New Run:

1. Name the run Baseline — production failure.
2. Select the dataset version containing the captured row.
3. Select the judge version that produced or represents the baseline decision.
4. Start the run and wait for the row’s judge result.

Verify that the baseline result matches the failure you observed. If it does not, calibrate the judge before using it as the regression criterion.

4. Run the candidate

Choose the path that matches the change:

• Judge change: start another platform run from the same config and dataset version, but select the candidate judge version.
• Agent, model, prompt, or tool change: run the agent fresh through the SDK with the same config. The SDK calls the candidate once per example and attaches its fresh trace so the judge scores the new execution rather than the saved production trace.

Name the run for the candidate change. Keep the dataset version fixed so the case roster does not move underneath the comparison. The current SDK run options are documented in the Python and TypeScript references.

5. Compare the results

Open Tests, select the baseline and candidate runs, and choose Compare. Filter to Different Result when the SDK reported pass conditions, or Different Judge Outputs otherwise.

Open the captured row and compare:

• The baseline production trace and candidate trace, when the agent changed
• The selected judge versions, when the evaluator changed
• The judge values, reasons, and cited evidence
• The example fields that define the regression case

The tutorial succeeds when the comparison shows whether the candidate fixed the known failure without changing the test input.

Keep the loop running

After release, monitor the behavior on new production traces. Add new failure shapes to the same dataset in a deliberate batch, review the new dataset version, and run the regression config again.

This is the complete loop:

failed production trace → dataset row → regression test → run comparison → production monitoring

Troubleshooting

• If the baseline judge result is wrong, calibrate the judge before comparing agent changes.
• If the candidate still shows the original trace, the run re-scored saved evidence. Use an SDK agent function for a fresh execution.
• If the comparison is unavailable, confirm that both runs use the same test config.
• If a newly added case is missing, verify that the run pinned the dataset version containing it.