Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Track judge labels

Track binary or classification judge outputs across production traffic.

A label is an output a binary or classification judge applies to a trace or session. It stays under its owning judge and uses that judge’s evaluation mode, sampling, scope, and triggers. Creating the judge does not enable online monitoring. Numeric outputs remain scores, not labels.

Create a judge for a stable signal you want to measure repeatedly, such as an unsupported claim, an incorrect retrieval, or a frustrated user. Keep broad exploration in Production issues until the failure pattern is clear enough to define as a judge criterion.

Choose the evaluation population

A useful judge applies the same criterion to a meaningful population:

  • Start with Lite for simple classifications. For more complex criteria, compare Normal or High on reviewed examples before increasing reasoning and cost.
  • Use a span trigger when only traces containing a particular tool or workflow are eligible. If the judge detects a missing operation, do not require that operation as a trigger.
  • Use trace scope when one agent run contains the necessary evidence.
  • Use session scope when the label depends on multiple related interactions.
  • Choose a sampling rate based on how many labeled traces the team needs.

For example, a Follow-up contains unsupported claims judge can require a messaging span, use trace scope, and evaluate every externally delivered message. A broad request-type judge can use a lower sampling rate when the goal is to estimate the traffic mix.

Track a judge label

This workflow inspects the judge’s choices, enables online evaluation, and verifies that the judge labels new traffic.

Prerequisites

  • An instrumented agent that is sending completed traces to the target project
  • A binary or classification judge that can evaluate those traces
  • Permission to edit the judge

For the steps below, use a classification judge named Response quality with the choices helpful, needs review, and unsafe. If it does not exist yet, follow Create an Agent judge, choose Classification, and add those three choices instead of the guide’s binary example.

Inspect the judge outputs

  1. Open Judges and select Response quality.
  2. In Rubric, confirm the classification choices and their descriptions.
  3. Confirm that helpful, needs review, and unsafe appear under the judge in the Judges list.

Each classification choice is a value the judge can return. A label is that value applied to a trace or session. Binary judges apply the label when the criterion is met (true). Numeric outputs remain scores. Selecting a value in the list opens the judge’s Labels tab.

Configure online evaluation

Open Config and enable Online monitoring. Use the controls below it to configure the traffic the judge should evaluate:

  • Use 100% sampling for this first verification. Lower it after the signal is working if you need to control evaluation volume.
  • Keep trace scoring unless the rubric needs the history of an entire session.
  • Add span triggers only when the judge should evaluate a narrower subset of traffic.

These settings belong to Response quality, not to an individual label. All three choices share the same mode, sampling, scope, and triggers.

Select Save, choose the proposed minor version, and select Publish to production. Confirm the production badge on the selected version. Choosing Create draft saves the new settings without applying them to production; promote that saved version when ready.

Get the judge to apply one known label

Run the instrumented agent once with a controlled test case that the judge should clearly label with one choice. For example, use a staging response that intentionally violates the unsafe criterion without including real secrets or harmful instructions.

Wait for the completed trace and asynchronous judge evaluation to reach Judgment. Continuous evaluation applies only to new eligible traffic; creating the judge does not prove that an evaluation has run.

Verify the online label

  1. Open Judges > Response quality.
  2. Open the Labels tab and select unsafe.
  3. Confirm that the new trace appears in the labeled traces table.
  4. Open it and verify that the label value is unsafe, the reason matches the trace evidence, and the label count and label rate include the new trace.

You now have one verified online label: the label is nested under its judge, the judge owns its online policy, and the label points back to the trace it was applied to.

Troubleshoot a missing label

  • Confirm that Online monitoring is enabled on the production version, rather than only on a saved draft.
  • Confirm that sampling is 100% for the verification run.
  • Confirm that the trace reached the same project and completed after you published the monitoring settings.
  • Remove or widen span triggers that exclude the test trace.
  • If session scoring is enabled, wait for the configured session completion condition.
  • Open the trace and check for an evaluation error before changing the rubric.

For missing or incomplete traces, follow Instrument your agent.

Next step

After the signal is stable, configure an automation or alert for the label. To reuse its labeled traces offline, add them to a dataset, then turn the failure into a regression test.

Was this page helpful?