Judges

What a judge does and where its labels show up.

A judge reads a trace or a session and applies one of two or more labels you define. You pick the Evaluation method when you create the judge, and it can’t change later:

• Agentic Judge: an Agent judge follows your rubric and reads the trace to decide. Most judges are Agent judges.
• Deterministic conditions: a Condition judge checks the trace’s duration, LLM cost, errors, or the names and attributes of its spans. It has exactly two labels: the first means the conditions matched, the second that they didn’t (Matched and Not matched by default). Use one for cost, latency and error alerts.
• Custom code: a Code judge runs Python you upload. You create and update Code judges through the API or MCP, not in the app. See Code judge bundle.

What judges are for

• Watch for a known failure. Turn the judge on for live traffic and every new trace gets a label.
• Alert your team. An action sends an email, posts to Slack, adds the trace to a dataset or runs another judge when the judge returns a given label.
• Grade tests. The same judges grade each trace when you run a test over a dataset.
• Index your data. Labels let the Judgment Agent answer questions across all your traces without reading every one. Labels as indexes explains how.

Where labels live

Labels are stored as judge results next to your traces. You can see them in three places:

• On the judge’s Labels tab, with the traces each label was applied to.
• In the trace view and as filters on Traces.
• In SQL. Judgment’s virtual SQL (vsql) exposes judge results in telemetry.scores and the label definitions in config.behaviors. API and SQL names still say behavior; it means label.

The Judgment Agent and your coding agent (through the MCP sql_query tool) both read these tables. See Query with SQL.

A judge has three tabs: Details (run settings, evaluation method and actions), Labels, and Alignment (see Align a judge).