--- title: "Evaluation data model" description: "Understand how judges, outputs, results, and behaviors represent evaluation in Judgment." sidebar: label: "Overview" seo: title: "Evaluation Data Model | Judgment Concepts" description: "Learn the relationship between Judgment judges, output types, evaluation results, and monitored behaviors." --- A **judge** is a reusable evaluation asset. It defines what to evaluate, how to evaluate it, and the shape of its output. You can apply the same judge to live traces, examples in a dataset, or an offline test. ```mermaid flowchart LR Evidence["Trace or example"] -->|"evaluated by"| Judge["Judge"] Judge -->|"defines"| Output["Output contract"] Output -->|"returned as"| Result["Result"] Output -->|"linked as"| Behavior["Behavior"] ``` ## Choose what the judge should explain Start with a question that helps a team understand or improve the agent. For a sales assistant, useful judges usually fall into three categories: | Category | Question | Example | | --- | --- | --- | | **User and request** | Who is using the agent, and what do they want? | Classify a request as meeting lookup, recap, preparation, synthesis, or follow-up. | | **Agent outcome** | Did the agent complete the requested task correctly? | Detect a follow-up containing claims unsupported by the retrieved meeting. | | **Environment** | Did the tools and data sources return what the agent needed? | Detect a failed meeting search or a transcript from the wrong meeting. | Keep agent outcomes separate from environment failures. If the assistant requests the wrong meeting, evaluate the agent outcome. If it requests the right meeting and the retrieval tool returns the wrong transcript, evaluate the environment. ## Follow one evaluation through the chain Suppose you create a categorical judge named **Response quality** with three possible outputs: - `helpful` - `needs review` - `unsafe` The judge is the reusable definition. Its categorical output contract limits every evaluation to one of those three values. When the judge evaluates one trace, it emits one **result**, such as `unsafe`, together with its reasoning and any trace evidence. You can then link the categorical outputs as **behaviors**. Judgment tracks the traces that receive each linked value, so `helpful`, `needs review`, and `unsafe` become measurable views under the **Response quality** judge. The judge can still exist with zero behaviors. For example, you can use it only on demand or in offline tests. Creating behaviors opts its binary or categorical outputs into the monitoring workflow. ## Judge method and output type answer different questions The judge's method determines how it evaluates evidence: - An **Agent Judge** follows a natural-language rubric and reasons over the supplied trace or example. - A **Code Judge** runs Python logic that you implement. Use an Agent Judge when the decision requires interpretation, such as whether a meeting brief includes the requested information. Use a Code Judge when the condition is exact, such as a timeout, exception, empty result, or known field value. The output type determines what every evaluation returns: | Output type | One result contains | Can become a behavior? | | --- | --- | --- | | Binary | `true` or `false` | Yes. The linked behavior identifies traces where the criterion is detected. | | Categorical | One configured choice, such as `unsafe` | Yes. Each linked choice is a behavior under the judge. | | Numeric | A number within the judge's configured range | No. Numeric outputs remain scores. | Changing from an Agent Judge to a Code Judge changes the evaluation method, not this output model. ## A result is one evaluation outcome A result belongs to one evaluation of a trace, session, or example. It records the judge, output value, reasoning, and evaluation context. Repeated evaluations produce separate results; the result is not the judge definition and is not itself a behavior. In a trace view, you inspect the result to understand why the judge returned its value. In an offline test, results let you compare judge and agent versions across a fixed dataset. ## A behavior is a monitored output A behavior is a binary or categorical output that you choose to track across online traffic. Behaviors live under their owning judge on the **Judges** page: - The Judges list shows behaviors indented beneath their owning judge. - Selecting a behavior opens the owning judge's **Behaviors** tab. - The tab shows detection rate, activity, matching traces, and result reasons. There is no separate behavior definition outside the judge. A categorical judge may own several behaviors, a binary judge may own one, and either may own none. ## Online settings belong to the judge Continuous versus on-demand evaluation, sampling, trace or session scope, and span triggers are configured once on the judge. Every behavior linked to that judge shares those online settings. This keeps all outputs from a categorical judge on the same traffic population and sampling policy. Configure them in this order: 1. Use span triggers to select traces containing the required operation or attribute. 2. Choose trace scope when one run contains the evidence, or session scope when earlier turns can change the decision. 3. Apply sampling after eligibility is defined. Use `100%` only when every eligible result is operationally important or needed for filtering. Create or edit these settings from the judge's **Trigger** controls. A numeric judge can still evaluate a live trace when explicitly applied, but it cannot be enabled as a continuously monitored behavior. Its values remain scores. ## Put the model into practice Define a natural-language rubric and use it in reusable evaluations. Implement a Python judge and inspect the result of a local evaluation run. Link a judge output, enable online evaluation, and verify one detection.