Judge actions
Send an email, post to Slack, save a trace to a dataset, or run more judges when a judge returns a given result.
An action runs after a judge evaluates a trace in production, based on the result the judge returned. The Alerting cookbook walks through a full example.
Actions react to one evaluation at a time. For trends or totals over a week, use a Scheduled Task instead.
Action groups
A judge’s actions live in action groups. Each group has its own result condition, environments, frequency and cooldown, and runs its actions when an evaluation matches. A judge can have several groups: for example, one that posts every Attempted result in production to Slack, and another that adds both results from staging to a dataset.
Add an action group
- Open the judge, then the Details tab, and expand Actions.
- Optionally give the group a Name.
- Select Add action and pick Send email, Notify on Slack, Add to dataset or Run judge. Fill in its destination. A group can hold several actions.
- Under When the result is, select the outputs that run the group’s actions, or Select all.
- Under In environments, pick the environments whose evaluations run the actions. Leave it on all environments to act everywhere.
- Optionally open Advanced to set frequency and cooldown.
- Save and publish the version.
For example, a jailbreak judge on the lunch-order agent with the outputs Attempted and Not attempted gets a group whose When the result is is Attempted, with a Notify on Slack action.
Once the judge has one group, Add action group adds another. To remove a group, open its menu and select Delete action group. Each group and each action has its own switch, so you can turn one off without deleting it.
Action groups are saved with the judge version. Actions run only after an automatic online evaluation of the production version, so turn on online monitoring under Run settings. Test runs and judges you run on selected traces never send actions. An evaluation that ends in an error doesn’t match any condition.
Judges created before GA may be Binary or Score. For a Binary judge, When the result is offers Any output, Detected and Not detected. For a Score judge it offers Any output and a Score comparison with a threshold.
You can also ask the Judgment Agent to add an action, or use the MCP tools: list_judge_action_groups and set_judge_action_groups, or the action configuration on create_judge and update_judge. See MCP tools.
Actions
| Action | What you configure |
|---|---|
| Send email | One Email address |
| Notify on Slack | The organization’s default channels or one channel, plus an optional message |
| Add to dataset | A dataset with one trace field (or one session field for a session judge) |
| Run judge | Other judges to run on the same trace |
Notify on Slack needs a Slack connection for your organization. See Connections.
By default, the action posts to the organization’s default Slack channels (Use organization default channels). To select one channel instead, clear that checkbox. If the organization has no default channels, you have to select one before you can save.
The message includes:
- The judge’s result and reasoning
- The trace’s input and output preview
- A button to open the trace
- Your Message (optional) text
Use Run judge to run an expensive judge only on traces a cheaper judge flagged first. The judges it starts run their production version on the same trace, and they do run their own actions.
Their own run settings don’t apply: online monitoring, environments, run conditions, and sampling rate. A judge that already ran earlier in the same chain is skipped, so two judges that run each other can’t loop.
Frequency and cooldown
Open Advanced in a group. The line under it summarizes the current limits.
- Run actions only if conditions matched at least: the actions run only once that many matching evaluations happen inside the window. The default is 1 time in the last 1 minute, so every match runs them.
- Once actions are triggered, wait: after the actions run, wait this long before running them again. The default is 0.
Both take a unit: seconds, minutes, hours or days. The limits apply to every enabled action in the group, so a cooldown set to quiet Slack also holds back Add to dataset in the same group. Put actions that need different limits in separate groups.
Alert on cost, duration or errors
Create a judge with Deterministic conditions as its evaluation method and add an action group to it. The conditions are Duration (seconds), LLM cost (USD), Error, and span groups that check a span’s Span Name or Span Attribute on the Root span or Any single span. Combine them with AND, OR and parentheses. The judge returns Matched or Not matched, so set When the result is to Matched.
If an action didn’t run
To confirm an action ran, check its destination: the inbox, the Slack channel, the dataset, or the other judge’s Labels tab. If nothing arrived, check that:
- The group and the action are both switched on.
- Online monitoring is on, and its environments, run conditions and sampling rate cover the trace.
- The version with the action is the production version.
- The result and environment match the group’s When the result is and In environments.
- The group’s frequency threshold was reached and it isn’t in its cooldown.
Automations
The Automations tab under Monitoring holds automations created before actions moved onto judges. Existing automations keep running for now. Configure new actions on a judge instead. Judgment will migrate supported automations into judge action groups before automations are turned off.