Behaviors

Track binary or categorical judge outputs across production traffic.

A behavior is a binary or categorical judge output selected for production monitoring. It stays under its owning judge and uses that judge’s evaluation mode, sampling, scope, and triggers. Numeric outputs remain scores rather than behaviors.

Create a behavior for a stable signal you want to measure repeatedly, such as an unsupported claim, an incorrect retrieval, or a frustrated user. Keep broad exploration in Production issues until the failure pattern is clear enough to define as a judge criterion.

Choose the evaluation population

A useful behavior measures the same criterion over a meaningful population:

Use a span trigger when only traces containing a particular tool or workflow are eligible.
Use trace scope when one agent run contains the necessary evidence.
Use session scope when the result depends on multiple related interactions.
Choose a sampling rate based on how many eligible results the team needs.

For example, a Follow-up contains unsupported claims behavior can require a messaging span, use trace scope, and evaluate every externally delivered message. A broad request-type behavior can use a lower sampling rate when the goal is to estimate the traffic mix.

Create and monitor a behavior

This workflow links judge outputs as behaviors, enables online evaluation, and verifies one detection from new traffic.

Prerequisites

An instrumented agent that is sending completed traces to the target project
A binary or categorical judge that can evaluate those traces
Permission to edit the judge

For the steps below, use a categorical judge named Response quality with the choices helpful, needs review, and unsafe. If it does not exist yet, follow Create an Agent Judge, choose Classification, and add those three choices instead of the guide’s binary example.

Link the outputs as behaviors

Open Judges and select Response quality.
In the judge’s Details panel, select Set up monitoring.
In Verify Configuration, confirm that helpful, needs review, and unsafe appear as the behavior options. Add descriptions or organization categories if they help teammates interpret the signals.

Each categorical option becomes a behavior under this judge. A binary judge creates one behavior for its detected condition. Numeric judge outputs cannot be linked as behaviors because they remain scores.

Configure online evaluation

In Advanced Settings, configure the traffic the judge should evaluate:

Keep the evaluation mode on Continuous.
Use 100% sampling for this first verification. Lower it after the signal is working if you need to control evaluation volume.
Keep trace scoring unless the rubric needs the history of an entire session.
Add span triggers only when the judge should evaluate a narrower subset of traffic.

These settings belong to Response quality, not to an individual behavior. All three linked behaviors share the same mode, sampling, scope, and triggers.

Select Update to link the behaviors and save the judge’s online settings.

Confirm the nested behavior rows

Return to the Judges list. Confirm that the three behaviors are indented beneath Response quality:

helpful
needs review
unsafe

Selecting a nested row opens the owning judge’s Behaviors tab. It does not open a separate Behaviors product area.

Produce one known detection

Run the instrumented agent once with a controlled test case that should clearly match one category. For example, use a staging response that intentionally violates the unsafe criterion without including real secrets or harmful instructions.

Wait for the completed trace and asynchronous judge evaluation to reach Judgment. Continuous evaluation applies only to new matching traffic; linking the behaviors does not prove that an evaluation has run.

Verify the online result

Open Judges > Response quality.
Open the Behaviors tab and select unsafe.
Confirm that the new trace appears in the matching traces table.
Open it and verify that the result value is unsafe, the reason matches the trace evidence, and the detection count or rate reflects the new result.

You now have one verified online detection: the behavior is nested under its judge, the judge owns its online policy, and the result points back to the trace that caused the detection.

Troubleshoot a missing detection

Confirm that the judge is Continuous, not On demand.
Confirm that sampling is 100% for the verification run.
Confirm that the trace reached the same project and completed after you saved the behavior.
Remove or widen span triggers that exclude the test trace.
If session scoring is enabled, wait for the configured session completion condition.
Open the trace and check for an evaluation error before changing the rubric.

For missing or incomplete traces, follow Instrument your agent.

Next step

After the signal is stable, configure an automation or alert for the behavior. To reuse its matching traces offline, add them to a dataset, then turn the failure into a regression test.