Agent Behavior Monitoring (ABM)

This page breaks down theoretical concepts of agent behavior monitoring (ABM). To get started with actually monitoring your agents, check out our monitoring docs!

When you're ready to deploy your agent, you need to be able to monitor its actions. While development and testing help catch many issues, the unpredictable nature of user inputs and non-deterministic agent behavior means that regressions are inevitable in production.

This is where monitoring becomes crucial - it's a window into how your agents interact with users, the most valuable data for improving your system.

Monitoring your agents helps you:

Track tool-use across your agent fleet in production, understanding how people use your system.

Catch and debug errors in real-time as they impact your customers, enabling quick response to issues.

Ensure system reliability by identifying patterns and risks before they affect multiple users.

Key Areas to Monitor

Collect agent telemetry in 30 seconds

Click here to collect all of the following data from your agent fleets with our tracing.

Agent Behavior Metrics

Use CustomScorers and PromptScorers with Online Evals to track key agent behavior metrics in real-time, such as:

Goal completion: Is the agent successfully completing the task or is it causing customer irritation?

Task alignment: Is the agent following instructions, using tools appropriately, or responding in a way that's helpful and contextually aware?

Correctness: Is the agent producing correct, domain-specific outputs?

Take action on your agents' behavior with rules and alerts.

Tool Usage Tracking

Tool usage telemetry can help you:

Identify performance bottlenecks/Optimize resource allocation (e.g. which tools might be overloaded)

Spot unusual patterns in tool selection (ex: tools that are rarely/never called)

Error Detection and Analysis

Real-world interactions can lead to various types of errors:

API failures and rate limits

Network timeouts

Resource constraints

Having real-time updates on errors can help you improve agent reliability by understanding common failure modes and addressing them through inspection of specific agent traces.

Learn More

To dive deeper into monitoring your agents, check out:

Online Evals for real-time alerts and actions on your agent's specific behavior
Rules to set up automated alerts based on eval results
Tracing to get started with tracking your agent's interactions