Read and compare test config runs

See how each test case did in a test config run, and what changed between two runs.

Every run belongs to a dataset, so you find it from the dataset.

• Open Datasets & Tests, then the dataset. The panel on the left lists its runs. Use Search tests to filter, and Group runs to group them by Test config, by Dataset version, or No grouping.
• On Datasets & Tests, the Test runs tab lists runs across every dataset with Run, Status, Test config, Dataset, Results, Judges and Created columns.
• The URL your code printed (result.ui_results_url in Python, result.uiResultsUrl in TypeScript) opens the run directly.

Read one run

The header shows when the run started and how long it took, the dataset version it used (for example v3), and how many judges graded it. Hover the judge count to see each judge’s version (for example v1.2) and how its labels or scores were spread across the run.

A run moves through Pending, Running, and then Completed, Error or Cancelled. A failed run shows an alert with the reason, such as “All judge evaluations failed”. If your agent raised errors, your terminal output from the run has them; the SDK logs each one and moves on.

The results table has one row per test case:

Use Display to hide judge columns you don’t need, or turn on Show judge reasoning to read why each judge decided what it did. Reasoning is the fastest way to tell a real agent failure from a judge that misread the trace.

Compare two runs

Compare runs of the same test config (same dataset, same judges), so any difference comes from what you changed.

1. In the runs panel, select two runs. A selection of runs from two different test configs can’t be compared; the button explains “Select two runs from the same test to compare.”
2. Select Compare.

You can also open one run and use Compare in its header, then pick the other run from the list.

The comparison has three tabs. When either run had a pass condition, they are All, Same Result and Different Result. Otherwise they are All, Same Judge Outputs and Different Judge Outputs. Start with the Different tab: it’s the short list of test cases where v2 behaves differently from v1. A test case that exists in only one of the runs counts as different.

If a judge graded the two runs with different versions, its column shows both, for example v1.1 ⇄ v1.2. In that case the difference may come from the judge, not your agent. Pin judge_versions in code to keep them equal; see Run a test config from your code.

Select Stop comparing to go back to a single run.

Cancel or delete runs

Select runs in the runs panel:

• Cancel stops runs that are still Pending or Running.
• Delete removes runs. It needs the admin role. Evaluation history and stored result rows are kept separately.

If your selection mixes active and finished runs, you get both buttons with counts, such as Cancel 1 and Delete 2.

Read results from code or a coding agent

The SDK returns the same results. In Python, result.results has one entry per test case, each with scorers_data (judge name, value, error, and success from your pass condition), and result.passed is True, False, or None when there was no pass condition. TypeScript returns result.results[].scorers and result.passed.

A coding agent connected over MCP can read runs with list_test_runs, get_test_run and get_test_run_items. See Connect your coding agent and MCP tools.