---
title: "Track judge labels"
description: "Track binary or classification judge outputs across production traffic."
sidebar:
  label: "Track judge labels"
seo:
  title: "Track Judge Labels | Judgment Docs"
  description: "Learn how judges apply labels to live traffic, then configure a judge and verify one labeled trace."
---

A label is an output a binary or classification judge applies to a trace or
session. It stays under its owning judge and uses that judge's evaluation mode,
sampling, scope, and triggers. Creating the judge does not enable online
monitoring. Numeric outputs remain scores, not labels.

<span id="link-the-outputs-as-behaviors" />

Create a judge for a stable signal you want to measure repeatedly, such as
an unsupported claim, an incorrect retrieval, or a frustrated user. Keep broad
exploration in [Production issues](/documentation/monitoring/production-issues)
until the failure pattern is clear enough to define as a judge criterion.

## Choose the evaluation population

A useful judge applies the same criterion to a meaningful population:

- Start with **Lite** for simple classifications. For more complex criteria,
  compare **Normal** or **High** on [reviewed examples](/documentation/tests/judge-calibration)
  before increasing reasoning and cost.
- Use a span trigger when only traces containing a particular tool or workflow
  are eligible. If the judge detects a missing operation, do not require that
  operation as a trigger.
- Use trace scope when one agent run contains the necessary evidence.
- Use session scope when the label depends on multiple related interactions.
- Choose a sampling rate based on how many labeled traces the team needs.

For example, a **Follow-up contains unsupported claims** judge can require a
messaging span, use trace scope, and evaluate every externally delivered
message. A broad request-type judge can use a lower sampling rate when the
goal is to estimate the traffic mix.

## Track a judge label

This workflow inspects the judge's choices, enables online evaluation, and
verifies that the judge labels new traffic.

### Prerequisites

- An instrumented agent that is sending completed traces to the target project
- A binary or classification judge that can evaluate those traces
- Permission to edit the judge

For the steps below, use a classification judge named **Response quality** with
the choices `helpful`, `needs review`, and `unsafe`. If it does not exist yet,
follow [Create an Agent judge](/documentation/judges/agent-judges), choose
**Classification**, and add those three choices instead of the guide's binary
example.

1. ### Inspect the judge outputs

    1. Open **Judges** and select **Response quality**.
    2. In **Rubric**, confirm the classification choices and their descriptions.
    3. Confirm that `helpful`, `needs review`, and `unsafe` appear under the judge
       in the **Judges** list.

    Each classification choice is a value the judge can return. A label is that
    value applied to a trace or session. Binary judges apply the label when the
    criterion is met (`true`). Numeric outputs remain scores. Selecting a value in the
    list opens the judge's **Labels** tab.

2. ### Configure online evaluation

    Open **Config** and enable **Online monitoring**. Use the controls below it
    to configure the traffic the judge should evaluate:

    - Use `100%` sampling for this first verification. Lower it after the signal is
      working if you need to control evaluation volume.
    - Keep trace scoring unless the rubric needs the history of an entire session.
    - Add span triggers only when the judge should evaluate a narrower subset of
      traffic.

    These settings belong to **Response quality**, not to an individual label.
    All three choices share the same mode, sampling, scope, and triggers.

    Select **Save**, choose the proposed minor version, and select **Publish to
    production**. Confirm the **production** badge on the selected version. Choosing
    **Create draft** saves the new settings without applying them to production;
    [promote that saved version](/documentation/judges/versions) when ready.

3. ### Get the judge to apply one known label

    Run the instrumented agent once with a controlled test case that the judge
    should clearly label with one choice. For example, use a staging response that
    intentionally violates the `unsafe` criterion without including real secrets or
    harmful instructions.

    Wait for the completed trace and asynchronous judge evaluation to reach
    Judgment. Continuous evaluation applies only to new eligible traffic; creating
    the judge does not prove that an evaluation has run.

4. <span id="verify-the-online-result" />

    ### Verify the online label

    1. Open **Judges > Response quality**.
    2. Open the **Labels** tab and select `unsafe`.
    3. Confirm that the new trace appears in the labeled traces table.
    4. Open it and verify that the label value is `unsafe`, the reason matches the
       trace evidence, and the label count and label rate include the new trace.

    You now have one verified online label: the label is nested under its
    judge, the judge owns its online policy, and the label points back to the
    trace it was applied to.

## Troubleshoot a missing label

- Confirm that **Online monitoring** is enabled on the production version,
  rather than only on a saved draft.
- Confirm that sampling is `100%` for the verification run.
- Confirm that the trace reached the same project and completed after you
  published the monitoring settings.
- Remove or widen span triggers that exclude the test trace.
- If session scoring is enabled, wait for the configured session completion
  condition.
- Open the trace and check for an evaluation error before changing the rubric.

For missing or incomplete traces, follow [Instrument your
agent](/documentation/tracing/instrumentation).

## Next step

After the signal is stable, [configure an automation or
alert](/documentation/monitoring/automations-and-alerts) for the label. To
reuse its labeled traces offline, [add them to a
dataset](/documentation/datasets/production-traces), then [turn the failure
into a regression test](/documentation/tests/regression-cases).