Production monitoring
Track known agent behaviors across live traffic and inspect the traces behind them.
Production monitoring applies known behaviors to live agent traffic. It shows how often a signal appears and keeps the matching traces available for investigation.
Choose the monitoring path
| Starting point | Use | Outcome |
|---|---|---|
| A known criterion | A judge output linked as a behavior | Rate, volume, and matching traces |
| One reported failure | Trace investigation | An evidence chain and saved regression case |
| An unknown recurring pattern | Production issues | A diagnosis with representative traces |
| A known condition requiring action | Automation | A notification, webhook, dataset action, or additional evaluation |
For example, monitor Unsupported follow-up when a judge already defines that failure. Use Production issues when meeting-retrieval failures appear to be increasing but you do not yet know the shared cause. Investigate the cited traces, then create a judge and behavior when the pattern becomes a stable criterion.
Monitor a behavior
Prerequisites
- An instrumented agent sending completed traces to the target project
- A binary or categorical judge with at least one behavior
- Continuous evaluation enabled for that judge
If the signal does not exist yet, follow Create and monitor a behavior first. Behavior creation and online evaluation settings belong to the owning judge.
Choose the behavior
Open Judges. Behaviors appear as indented rows beneath their owning judge.
Choose the row for the value you want to monitor, such as unsafe beneath a
categorical Response quality judge.
The behavior opens in the judge’s Behaviors tab. If the judge has several categorical behaviors, use the tabs to switch between values without leaving the judge.
Set the time window
Choose a time range that matches the question you are asking. Use a short window to validate a new signal or investigate a recent deployment, and a longer window to understand its baseline.
Keep the same time window while reading the rate, volume, and matching traces. Changing it between views makes the numbers answer different questions.
Read the rate and volume together
The behavior row and detail view show two different measures:
- Detection rate is the share of scored traces where this behavior fired.
- Trace count is the number of traces assigned to this behavior in the selected window.
Interpret the rate using the behavior’s meaning. A higher helpful rate is
usually good; a higher unsafe rate is usually bad. The name alone does not
establish whether an increase is a regression.
Sampling also matters. These measures describe traces the judge evaluated, not necessarily every trace sent to the project.
Inspect matching traces
Use the traces table below the behavior statistics to find representative detections. Open a trace and verify:
- The judge returned the behavior value you selected.
- The reason points to evidence in the trace.
- Any cited span, input, output, or tool call supports that reason.
- The result did not contain an evaluation error.
Use filters to narrow by customer, session, deployment, or another trace attribute when you need to localize the signal. Follow Investigate a failed trace for a complete trace investigation path.
Decide how to act
Once the signal is trustworthy, choose the response that fits it:
- Configure an automation or alert when a matching trace should notify a team or trigger an action.
- Add representative traces to a dataset when they should become regression cases.
- Refine the judge when the result reasons do not match the evidence.
Choose one next action after the behavior’s results match the trace evidence.
Continue with Automations and alerts when the behavior should trigger a response.
Verify the monitoring loop
Send one controlled trace that should clearly match the behavior. Confirm that it appears in the behavior’s traces table, that the result reason is correct, and that the detection count changes for the selected time window.
If it does not appear, confirm that the judge is Continuous, sampling includes the test trace, the trace completed after monitoring was enabled, and the trace reached the same project. For a missing trace, see Instrument your agent.