Evaluation data model
Understand how judges, outputs, results, and behaviors represent evaluation in Judgment.
A judge is a reusable evaluation asset. It defines what to evaluate, how to evaluate it, and the shape of its output. You can apply the same judge to live traces, examples in a dataset, or an offline test.
Choose what the judge should explain
Start with a question that helps a team understand or improve the agent. For a sales assistant, useful judges usually fall into three categories:
| Category | Question | Example |
|---|---|---|
| User and request | Who is using the agent, and what do they want? | Classify a request as meeting lookup, recap, preparation, synthesis, or follow-up. |
| Agent outcome | Did the agent complete the requested task correctly? | Detect a follow-up containing claims unsupported by the retrieved meeting. |
| Environment | Did the tools and data sources return what the agent needed? | Detect a failed meeting search or a transcript from the wrong meeting. |
Keep agent outcomes separate from environment failures. If the assistant requests the wrong meeting, evaluate the agent outcome. If it requests the right meeting and the retrieval tool returns the wrong transcript, evaluate the environment.
Follow one evaluation through the chain
Suppose you create a categorical judge named Response quality with three possible outputs:
helpfulneeds reviewunsafe
The judge is the reusable definition. Its categorical output contract limits
every evaluation to one of those three values. When the judge evaluates one
trace, it emits one result, such as unsafe, together with its reasoning
and any trace evidence.
You can then link the categorical outputs as behaviors. Judgment tracks the
traces that receive each linked value, so helpful, needs review, and
unsafe become measurable views under the Response quality judge.
The judge can still exist with zero behaviors. For example, you can use it only on demand or in offline tests. Creating behaviors opts its binary or categorical outputs into the monitoring workflow.
Judge method and output type answer different questions
The judge’s method determines how it evaluates evidence:
- An Agent Judge follows a natural-language rubric and reasons over the supplied trace or example.
- A Code Judge runs Python logic that you implement.
Use an Agent Judge when the decision requires interpretation, such as whether a meeting brief includes the requested information. Use a Code Judge when the condition is exact, such as a timeout, exception, empty result, or known field value.
The output type determines what every evaluation returns:
| Output type | One result contains | Can become a behavior? |
|---|---|---|
| Binary | true or false |
Yes. The linked behavior identifies traces where the criterion is detected. |
| Categorical | One configured choice, such as unsafe |
Yes. Each linked choice is a behavior under the judge. |
| Numeric | A number within the judge’s configured range | No. Numeric outputs remain scores. |
Changing from an Agent Judge to a Code Judge changes the evaluation method, not this output model.
A result is one evaluation outcome
A result belongs to one evaluation of a trace, session, or example. It records the judge, output value, reasoning, and evaluation context. Repeated evaluations produce separate results; the result is not the judge definition and is not itself a behavior.
In a trace view, you inspect the result to understand why the judge returned its value. In an offline test, results let you compare judge and agent versions across a fixed dataset.
A behavior is a monitored output
A behavior is a binary or categorical output that you choose to track across online traffic. Behaviors live under their owning judge on the Judges page:
- The Judges list shows behaviors indented beneath their owning judge.
- Selecting a behavior opens the owning judge’s Behaviors tab.
- The tab shows detection rate, activity, matching traces, and result reasons.
There is no separate behavior definition outside the judge. A categorical judge may own several behaviors, a binary judge may own one, and either may own none.
Online settings belong to the judge
Continuous versus on-demand evaluation, sampling, trace or session scope, and span triggers are configured once on the judge. Every behavior linked to that judge shares those online settings. This keeps all outputs from a categorical judge on the same traffic population and sampling policy.
Configure them in this order:
- Use span triggers to select traces containing the required operation or attribute.
- Choose trace scope when one run contains the evidence, or session scope when earlier turns can change the decision.
- Apply sampling after eligibility is defined. Use
100%only when every eligible result is operationally important or needed for filtering.
Create or edit these settings from the judge’s Trigger controls. A numeric judge can still evaluate a live trace when explicitly applied, but it cannot be enabled as a continuously monitored behavior. Its values remain scores.