Align a judge

Check a judge's labels against your own calls, then turn the disagreements into a better rubric.

Alignment works on judges that have already labeled some traces. It isn’t available for older Score judges.

Review the judge’s calls

Open Judges, pick the judge, and open the Alignment tab. The badge on the tab shows the current agreement rate.
Select Start review. In Start a Review Session, filter the traces if you want a particular slice (one environment, one customer, a time window), select the ones to review, and start.
For each trace, read the judge’s label and reasoning, then check the spans it cites. Switch to the session view when the judge scores whole sessions.
Choose Agree if the label is right. Choose Disagree if it isn’t, and pick the label it should have applied. Add a comment when the reason isn’t obvious, like “the user asked for 30 people and the order only covered 24.”
Select Next to move on, and End Review on the last trace. You’ll see a summary of how many you agreed and disagreed with.

Read the evidence the judge cites as well as its label. If you can’t tell from the trace whether the judge was right, the rubric probably can’t either.

Use the disagreements

Back on the Alignment tab, the table lists every review. Filter it with All reviews, Disagreed or Agreed.

Read the disagreements together and look for the pattern. With the lunch-order agent, a Request completed judge might keep labeling sessions Completed when the agent silently dropped Cedar Grill after two 504 Gateway Timeout errors. Then the rubric needs a line about options the agent drops without telling the user.

Select the reviews you want to act on, then:

Improve rubric opens a Judgment Agent chat with those reviews attached and a prompt, ready to send, that asks for targeted rubric changes. Nothing is published until you save. Save the change as a draft with Create draft, check it, then Publish to production.
Add to Chat adds the reviews to your current Judgment Agent chat, so you can ask about them directly.
Create golden dataset saves the reviewed traces (or sessions) with your labels as a dataset. Each trace appears once, even if you reviewed it more than once. Use it to test the judge or your agent against calls you’ve already made.
Remove deletes the selected reviews. This can’t be undone.

Keep going

Review a fresh batch after each rubric change and after anything that changes your traffic, like a new agent version or a new customer. The agreement badge on the tab, and the Alignment sort on the Judges list, show which judges need attention.

For Code judges, alignment shows you where the code is wrong, but the fix is in your code: update it and upload a new version. See Code judge bundle.