Calibrate a judge

Review a binary or categorical judge against human judgment and use disagreements to refine it.

Calibration tests whether a judge agrees with human judgment. You will review real outputs, record disagreements, and leave with a focused set of evidence for the next judge revision.

Prerequisites

A saved binary or categorical judge
Completed traces that the judge has already scored
A reviewer who can decide the correct outcome from the trace evidence

The current interactive Alignment review does not start for numeric judges. Validate numeric score criteria with a labeled dataset and offline comparisons instead.

Review the judge

Open Alignment

Open Judges, choose the judge, and select Alignment. The Past Reviews table shows earlier human decisions and whether they agreed with the judge.

Select Start Review. Filter the available traces when you need a particular failure mode, deployment, customer, or time window. Select individual rows, or leave the selection empty to review every visible trace, then start the review.

Inspect the evidence

For each trace, read the judge output and reason, then inspect the cited span, input, output, and tool evidence. Switch to the session view when the judge evaluates a whole session.

Do not vote from the output label alone. Calibration measures whether the result is supported by the evidence the judge actually received.

Record human judgment

Choose Agree when the judge returned the correct value. Choose Disagree when it did not; for categorical output, select the value the judge should have returned. Add a concise reviewer comment when it explains the missing criterion or misleading evidence.

Continue until the selected traces are reviewed, then choose End Review.

Use the disagreements

Read the review summary and return to Past Reviews. Filter to Disagreed and group the misses by a concrete cause: ambiguous rubric, missing context, incorrect category boundary, or unsupported reasoning.

Update the judge prompt or code, save a new version, produce new judge results on representative traces, and review those outputs again. Keep the prior version available as the baseline.

Verify calibration

The Alignment table should show the reviewed trace, Judge Output, Human Review, and an agreed or disagreed state. Calibration is useful when each disagreement points to a specific change you can make or a test case you should preserve—not merely when the displayed percentage moves.

Create a labeled dataset

To preserve human labels as reusable test data, select Label Dataset from the judge’s Alignment tab. Choose a trace-backed source dataset, name the new labeled dataset, and review each trace. Judgment creates a new dataset with the labels rather than mutating the source dataset.

Use that labeled dataset in offline tests when you need a stable calibration set across judge versions.

Troubleshooting

If Start Review is absent, confirm that the judge is binary or categorical and has been saved.
If no traces appear, the judge may not have scored any matching completed traces yet. Run it on representative evidence first.
If a trace has no judge output, treat it as missing evaluation data rather than automatically disagreeing.
If Label Dataset finds no source datasets, add a trace field and trace-backed rows to a non-legacy dataset.
If alignment improves only on the reviewed cases, add unreviewed holdout examples before concluding that the judge generalized.

Next step

Run an offline test to create baseline and candidate runs from one config. After both runs complete, compare the test runs over the same fixed dataset.