Compare test runs
Compare two runs from the same test config and localize result changes to examples, traces, and judge versions.
Compare runs to understand what changed between a baseline and a candidate. You will open two runs from the same config, isolate changed rows, and connect each difference to its input, trace, and judge versions.
Prerequisites
- Two test runs from the same test config
- Completed results for the rows and judges you need to compare
- Names that identify the baseline and candidate change
Runs from different configs cannot be compared because they do not share one dataset-and-judge binding.
Compare the runs
Create the candidate run
Reuse the baseline config and select New Run. Name the run for the candidate, then choose the dataset and judge versions deliberately.
For an agent change, use an SDK agent function so the candidate produces a fresh trace for every dataset row. For a judge change, keep the dataset version fixed and select the new judge version.
Open comparison
In Tests, select exactly two runs from the same config and choose Compare. You can also open one run, select Compare beside its title, and choose another run of the same config.
Isolate changed rows
Use the comparison tabs:
- Same Result and Different Result compare per-example pass/fail when SDK runs reported a pass condition.
- Same Judge Outputs and Different Judge Outputs compare judge values when no pass condition exists.
Switch between Stacked and Side by side layouts based on the amount of evidence you need to scan.
Inspect one difference
Open a changed row. Compare the example fields and saved or generated trace evidence, then read each judge value and reason in the analysis panel.
Check the run list and judge badges for the exact dataset and judge versions. A dataset membership change, agent execution change, and judge version change answer different questions.
Decide the outcome
Record whether the candidate fixed the target failures, introduced new disagreements, or changed only the evaluator. Inspect unchanged successes as well as changed failures before deciding to release.
Verify the comparison
The comparison header names both runs, the filters account for all visible rows, and every claimed improvement can be traced to one example plus its two sets of evidence and judge outputs.
Troubleshooting
- If Compare is disabled, select exactly two runs from the same config.
- If a candidate run is still pending or running, wait for the required judge results before interpreting it as unchanged.
- If rows are missing on one side, confirm that both runs used the intended dataset version.
- If Result is absent, the runs did not report an SDK pass condition. Use judge-output differences instead.
- If every trace-backed row is identical, remember that a platform run re-scores saved evidence; it does not execute a changed agent. Use a fresh agent trace through the SDK to test agent behavior.
Next step
When a production failure should stay in every release check, turn it into a regression test.