Bucketing
Set up bucketing to automatically route traces to datasets based on agent behavior.
Go from a bug report to a behavior bucket in seconds. With Bucketing, you can define and track agent behaviors, turning vague issues into clear, data-driven groupings of traces. Once you've bucketed behaviors, dive in to explore patterns and identify root causes.
Bucketing uses 3 core Judgment features:
TracePromptScorer: Evaluates full traces or subtrees based on your criteria.Dataset: Stores grouped traces that match your bucketing rules.Rules: Automatically adds traces to datasets when conditions are met.
Getting Started with Bucketing
Navigate to Bucketing Page
Click Bucketing in the sidebar under Project Configuration.
Describe Your Agent
Provide a description of your agent's purpose, capabilities, and typical tasks. This significantly helps improve bucketing accuracy by giving context about expected behaviors.
Click Save when you're done.
Create a Bucketing Configuration
A bucketing configuration defines the criteria for automatically routing traces to a dataset in real time based on specific behaviors or patterns.
Click New Configuration and fill in the name, description, and classification method. We recommend selecting Rubric to create a fine-tuned scorer through guided steps, or you can choose an existing TracePromptScorer.
Rubric Generation
You can review sample traces and mark them as Accept or Reject based on fit. This feedback generates suggestions for improving your rubric.
Accept or reject each suggested improvement based on what you're looking for. Once you're done reviewing, click Next.
Review and edit the final rubric to make sure it fits your needs, then name it. Click Next when you're satisfied.
Finalize Your Configuration
Review the names you've chosen for your configuration, dataset, and TracePromptScorer. Click Create Configuration to save. You can view all your configurations from the Bucketing page.
Once you've set it up, bucketing runs automatically in the background. Traces matching your criteria are added to the dataset in real time.
Next Steps
- Alerts - Configure alerts triggered on agent behavior to catch issues before they impact users.
- Tracer SDK Reference - Explore the complete Tracer API including span access, metadata, and advanced configuration.
- Agent Behavior Monitoring - Run real-time behavioral monitoring on your production agents with server-hosted scorers.