Automations and Alerts

Automations define conditions on evaluation metrics. When conditions are met, automations trigger alerts that can send notifications or execute actions.

Overview

Automations monitor metrics including:

Trace Attributes (Duration, LLM Cost, Exception, Span Attribute)
Triggered Behaviors

Each automation has conditions that determine when to trigger alerts. You can configure alert frequency thresholds and cooldown periods to control when notifications or actions execute.

Automations do not support local Code Judges. Upload Code Judges to the server before using them in automations. See Online Behavioral Monitoring.

Create an Automation

Create automations from your project's Automations section by clicking "New Automation".

Automation Configuration

Name: Descriptive name (required)
Description: Optional purpose explanation

Filter Conditions

Define when the automation triggers:

Match Type: "AND" (all conditions) or "OR" (any condition)
Conditions: Add one or more conditions with:
- Metric: Depends on trigger type:
  - Trace Attributes: Duration, LLM Cost, Exception, or custom Span Attribute
  - Behaviors: Behavior triggered
- Operator: Depends on trigger type:
  - Trace Attributes: Comparison (>=, <=, ==, <, >) for numeric metrics; exists, equals, contains for Exception and Span Attribute
  - Behaviors: detected
- Value: Threshold value (comparison operators only)

You can add multiple conditions with the "Add trigger" button.

Actions and Notifications

Configure what executes when alert frequency is met and cooldown has expired:

Below are the available actions and notifications:

Behavior Evaluation

Run behavior judges on the traces that trigger this automation. Select one or more judges to evaluate each matching trace.

You can optionally set a sampling rate (0–100%) to evaluate only a fraction of matching traces. For example, setting 25% evaluates roughly one in four traces. If left blank, all matching traces are evaluated (equivalent to 100%). Sampling uses a deterministic hash of the trace ID so the same trace is always included or excluded consistently.

/add-mention [organization-id] [@user_id] — Configure a user to ping when a notification is sent
/remove-mention [organization-id] [@user_id] — Remove a user from being pinged
/list-mentions [organization-id] — List all users being pinged by notifications
/add-channel [organization-id] — Configure the current channel to receive notifications
/remove-channel [organization-id] — Stop the current channel from receiving notifications
/list-channels [organization-id] — List channels configured to receive notifications

A workspace can be configured to receive notifications from multiple organizations. organization-id must be provided in each of these commands to specify the relevant organization.