Alerts
Set up automated alerts and actions based on evaluation metrics.
Rules allow you to define specific conditions for your evaluation metrics that can trigger alerts and actions when met. They serve as the foundation for the alerting system and help you monitor your AI system's performance against predetermined thresholds.
Overview
A rule consists of one or more conditions, each tied to a specific metric that is supported by our built-in scorers (like Faithfulness or Answer Relevancy). When evaluations are performed, the rules engine checks if the measured scores satisfy the conditions set in your rules. Based on the rule's configuration, alerts can be triggered and actions executed through various channels.
Rules are created through the monitoring section of your project. To create a new rule:
- Navigate to the Monitoring section in your project dashboard
- Click "Create New Rule" or access the rules configuration
- Configure the rule settings as described below
Rule Configuration
Basic Information
- Rule Name: A descriptive name for your rule (required)
- Description: Optional description explaining the rule's purpose
Filter Conditions
The filter section allows you to define when the rule should trigger. You can:
- Match Type: Choose between "AND" (all conditions must be met) or "OR" (any condition can trigger the rule)
- Conditions: Add one or more conditions, each specifying:
- Metric: Select from available built-in scorers (e.g., Faithfulness, Answer Relevancy)
- Operator: Choose comparison operator (
>=,<=,==, etc.) - Value: Set the threshold value
You can add multiple conditions by clicking "Add condition" to create complex rules.
The metric dropdown includes various built-in scorers you can choose from:
Actions
Configure what happens when rule conditions are met:
Add to Dataset
- Automatically add failing evaluations to a dataset for further analysis
- Select your target dataset from the dropdown menu
- Only trace datasets are supported for automated rule actions
Notification Actions Configure what actions should be taken when rule conditions are met:
Email Notifications
- Email: Send notifications to specified email addresses
- Add multiple email addresses for broader notification coverage
Slack Integration
- Slack: Post alerts to Slack channels through app integration
- Connect your Slack workspace through the App Integrations section
- Once connected, you can select channels
When configuring Slack in your rule actions, you'll see the connection status:
PagerDuty Integration
- PagerDuty: Create incidents in PagerDuty for critical issues
- Configure integration keys
- Set incident severity levels
Managing Rules
Once created, rules can be managed through the rules dashboard:
- Add Rules: Add new rules
- Edit Rules: Modify existing rule conditions and actions
- Delete Rules: Remove rules that are no longer needed
