---
title: "Compare test runs"
description: "Compare two runs from the same test config and localize result changes to examples, traces, and judge versions."
sidebar:
  label: "Compare test runs"
seo:
  title: "Compare Test Runs | Judgment How-to"
  description: "Select two compatible offline test runs, filter changed outcomes, and inspect side-by-side evidence in Judgment."
---

Compare runs to understand what changed between a baseline and a candidate.
You will open two runs from the same config, isolate changed rows, and connect
each difference to its input, trace, and judge versions.

## Prerequisites

- Two test runs from the same test config
- Completed results for the rows and judges you need to compare
- Names that identify the baseline and candidate change

Runs from different configs cannot be compared because they do not share one
dataset-and-judge binding.

## Compare the runs

1. ### Create the candidate run

    Reuse the baseline config and select **New Run**. Name the run for the
    candidate, then choose the dataset and judge versions deliberately.

    For an agent change, use an SDK agent function so the candidate produces a
    fresh trace for every dataset row. For a judge change, keep the dataset
    version fixed and select the new judge version.

2. ### Open comparison

    In **Tests**, select exactly two runs from the same config and choose
    **Compare**. You can also open one run, select **Compare** beside its title,
    and choose another run of the same config.

3. ### Isolate changed rows

    Use the comparison tabs:

    - **Same Result** and **Different Result** compare per-example pass/fail
      when SDK runs reported a pass condition.
    - **Same Judge Outputs** and **Different Judge Outputs** compare judge
      values when no pass condition exists.

    Switch between **Stacked** and **Side by side** layouts based on the amount
    of evidence you need to scan.

4. ### Inspect one difference

    Open a changed row. Compare the example fields and saved or generated trace
    evidence, then read each judge value and reason in the analysis panel.

    Check the run list and judge badges for the exact dataset and judge
    versions. A dataset membership change, agent execution change, and judge
    version change answer different questions.

5. ### Decide the outcome

    Record whether the candidate fixed the target failures, introduced new
    disagreements, or changed only the evaluator. Inspect unchanged successes
    as well as changed failures before deciding to release.

## Verify the comparison

The comparison header names both runs, the filters account for all visible
rows, and every claimed improvement can be traced to one example plus its two
sets of evidence and judge outputs.

## Troubleshooting

- If **Compare** is disabled, select exactly two runs from the same config.
- If a candidate run is still pending or running, wait for the required judge
  results before interpreting it as unchanged.
- If rows are missing on one side, confirm that both runs used the intended
  dataset version.
- If **Result** is absent, the runs did not report an SDK pass condition. Use
  judge-output differences instead.
- If every trace-backed row is identical, remember that a platform run
  re-scores saved evidence; it does not execute a changed agent. Use a fresh
  agent trace through the SDK to test agent behavior.

## Next step

When a production failure should stay in every release check, [turn it into a
regression test](/documentation/tests/regression-cases).