Monitoring

This page breaks down theoretical concepts of agent monitoring. To get started with actually monitoring your agents, check out our monitoring docs!

When you're ready to deploy your agent, you need to be able to monitor its actions. While development and testing help catch many issues, the unpredictable nature of user inputs and non-deterministic agent behavior means that regressions are inevitable in production.

This is where monitoring becomes crucial - it's a window into how your agents interact with users, the most valuable data for improving your system.

Monitoring your agents helps you:

Track tool-use across your agent fleet in production, understanding how people use your system.

Catch and debug errors in real-time as they impact your customers, enabling quick response to issues.

Ensure system reliability by identifying patterns and risks before they affect multiple users.

Key Areas to Monitor

Collect agent telemetry with 2 lines of code

Click here to collect all of the following data from your agent fleets with our tracing.

Tool Usage Tracking

Tool usage telemetry can help you:

Identify performance bottlenecks/Optimize resource allocation (e.g. which tools might be overloaded)
Spot unusual patterns in tool selection (ex: tools that are rarely/never called)

Error Detection and Analysis

Real-world interactions can lead to various types of errors:

API failures and rate limits
Network timeouts
Resource constraints

Having real-time updates on errors can help you improve agent reliability by understanding common failure modes and addressing them through inspection of specific agent traces.

Latency

Latency telemetry can help you:

Identify performance bottlenecks (e.g. which tools might be slow)
Spot unusual patterns in latency (ex: agent becomes consistently slow after a feature update)

Cost

Cost telemetry can help you:

Identify cost outliers (e.g. which tools might be expensive)
Spot unusual patterns in cost (ex: tools that are consistently expensive)

Learn More

To dive deeper into monitoring your agents, check out:

Tracing to get started with monitoring your agent's behavior
Online Evals for real-time alerts
Annotations for manual review and feedback of specific agent runs