Quickstart
What Judgment does for your agent, and how to set up your first project.
Judgment works from your agent’s traces, whether they come from production or from tests in your own environment.
Discover with Issues
Agents search your traces for recurring problems you didn’t know to look for.
Monitor with judges
A judge labels traces for a failure or signal you already know to look for, and can alert your team when it fires. Judges are agents, so they can evaluate long traces.
Analyze with the Judgment Agent
Judge labels act as indexes. The agent uses them to answer questions across all your data and cites the traces behind each answer.
Test before you ship
Tests run your agent over a set of test inputs and grade each trace with the same judges.
Confirm the fix
The same judges keep labeling new traces, so you can see whether the failure stops.
Before you start
- A project is one agent. Development, staging and production traffic live inside it as environments.
- You need a Judgment API key and your organization ID. Both are separate from your model provider’s key.
- The setup creates an organization ingest API key for you. It can only send traces.
- Only Admins and Owners can set up tracing. Developers can close the setup and ask one of them to do it.
Set up your project
When you create a project, the project home opens a five-step setup. Your coding agent does the tracing work, and Judgment starts finding Issues and creating judges in the background while you continue.
Tracing: Hand this off to your coding agent
- Copy the
.envblock with your new ingest API key and organization ID. The key is only shown once. If you lost it, select Create key to name and create another one. - Copy the prompt and paste it into your coding agent. It connects the Judgment MCP server, checks the connection, and sets up tracing in your agent.
- Run your agent. The step moves on by itself once your first trace arrives.
Issues: Find recurring problems
Judgment starts a first detection run over the traces that arrive (how detection works), then runs it daily. Select Got it, continue.
Judges: Track what matters in every trace
An agent also analyzes your traces and creates judges with online monitoring on, so new traces start getting labels right away. Select Got it, continue.
Slack: Keep your team in the loop
- Connect your workspace. Only Admins and Owners can connect it.
- Pick the channel that gets an issue report after each detection run, then type
/invite @Judgmentin that channel.
You can also ping @Judgment in Slack to ask about your traces.
You can skip this step and connect Slack later. Your team then checks the Issues page instead.
Judgment Agent: Ask about anything in your project
The chat opens with a question already filled in: it asks the agent where your agent struggled and to cite the traces behind each pattern. Send it, then follow the citations back to the traces.
While the first Issues run and judge creation are working, their progress shows on the project home.
What you can do next
The cookbooks show best practices for using Judgment. They all build from the same Example agent.
- Start with Issues, which reports recurring problems judges aren’t tracking yet. Root-cause an Issue takes one Issue from the inbox to a fix and a judge that watches for it.
- To analyze your agent data, write a few judges. Labels as indexes explains how, and Data analysis walks through the setup.
- To get alerted, add an action to a judge that sends an email or a Slack message when it applies a label. For cost and duration alerts, use a Condition judge with an action. Alerting builds both.
- To test a change before you ship it, see Tests. Testing compares two versions of the example agent and gates pull requests in CI.
- If you work in Codex, Claude Code, or Cursor, connect your coding agent over MCP. It can read your Issues, traces, and labels while it works, and fix an Issue you hand it.