---
title: "Calibrate a judge"
description: "Review a binary or categorical judge against human judgment and use disagreements to refine it."
sidebar:
  label: "Calibrate a judge"
seo:
  title: "Calibrate a Judge | Judgment How-to"
  description: "Review judge outputs on real traces, record agreement or the correct answer, and verify alignment in Judgment."
---

Calibration tests whether a judge agrees with human judgment. You will review
real outputs, record disagreements, and leave with a focused set of evidence
for the next judge revision.

## Prerequisites

- A saved binary or categorical judge
- Completed traces that the judge has already scored
- A reviewer who can decide the correct outcome from the trace evidence

The current interactive Alignment review does not start for numeric judges.
Validate numeric score criteria with a labeled dataset and offline comparisons
instead.

## Review the judge

1. ### Open Alignment

    Open **Judges**, choose the judge, and select **Alignment**. The **Past
    Reviews** table shows earlier human decisions and whether they agreed with
    the judge.

2. ### Select traces to review

    Select **Start Review**. Filter the available traces when you need a
    particular failure mode, deployment, customer, or time window. Select
    individual rows, or leave the selection empty to review every visible
    trace, then start the review.

3. ### Inspect the evidence

    For each trace, read the judge output and reason, then inspect the cited
    span, input, output, and tool evidence. Switch to the session view when the
    judge evaluates a whole session.

    Do not vote from the output label alone. Calibration measures whether the
    result is supported by the evidence the judge actually received.

4. ### Record human judgment

    Choose **Agree** when the judge returned the correct value. Choose
    **Disagree** when it did not; for categorical output, select the value the
    judge should have returned. Add a concise reviewer comment when it explains
    the missing criterion or misleading evidence.

    Continue until the selected traces are reviewed, then choose **End Review**.

5. ### Use the disagreements

    Read the review summary and return to **Past Reviews**. Filter to
    **Disagreed** and group the misses by a concrete cause: ambiguous rubric,
    missing context, incorrect category boundary, or unsupported reasoning.

    Update the judge prompt or code, save a new version, produce new judge
    results on representative traces, and review those outputs again. Keep the
    prior version available as the baseline.

## Verify calibration

The Alignment table should show the reviewed trace, **Judge Output**, **Human
Review**, and an agreed or disagreed state. Calibration is useful when each
disagreement points to a specific change you can make or a test case you should
preserve—not merely when the displayed percentage moves.

## Create a labeled dataset

To preserve human labels as reusable test data, select **Label Dataset** from
the judge's Alignment tab. Choose a trace-backed source dataset, name the new
labeled dataset, and review each trace. Judgment creates a new dataset with the
labels rather than mutating the source dataset.

Use that labeled dataset in offline tests when you need a stable calibration
set across judge versions.

## Troubleshooting

- If **Start Review** is absent, confirm that the judge is binary or
  categorical and has been saved.
- If no traces appear, the judge may not have scored any matching completed
  traces yet. Run it on representative evidence first.
- If a trace has no judge output, treat it as missing evaluation data rather
  than automatically disagreeing.
- If **Label Dataset** finds no source datasets, add a `trace` field and
  trace-backed rows to a non-legacy dataset.
- If alignment improves only on the reviewed cases, add unreviewed holdout
  examples before concluding that the judge generalized.

## Next step

[Run an offline test](/documentation/tests/offline-tests) to create baseline
and candidate runs from one config. After both runs complete, [compare the test
runs](/documentation/tests/test-run-comparison) over the same fixed dataset.