Quickstart

What Judgment does for your agent, and how to set up your first project.

Judgment works from your agent’s traces, whether they come from production or from tests in your own environment.

Discover with Issues

Agents search your traces for recurring problems you didn’t know to look for.

Monitor with judges

A judge labels traces for a failure or signal you already know to look for, and can alert your team when it fires. Judges are agents, so they can evaluate long traces.

Analyze with the Judgment Agent

Judge labels act as indexes. The agent uses them to answer questions across all your data and cites the traces behind each answer.

Test before you ship

Tests run your agent over a set of test inputs and grade each trace with the same judges.

Confirm the fix

The same judges keep labeling new traces, so you can see whether the failure stops.

Before you start

A project is one agent. Development, staging and production traffic live inside it as environments.
You need a Judgment API key and your organization ID. Both are separate from your model provider’s key.
Use an organization ingest API key. An organization Admin creates it under Settings > Ingest API Keys, and it can only send traces. A personal key from Settings > Personal API Keys also works.
Only Admins and Owners see the Tracing step. Developers should ask one of them to set it up.

Set up your project

When you create a project, the project home opens a five-step setup. If you close it, reopen it from Project guide in the Support menu.

Tracing: Set up tracing

Select Create an ingest API key, or use one you already saved. Copy the value when it appears.
Under Prompt your coding agent, pick a tab:
- Coding agent: copy the prompt, or open it in Codex, Claude Code, Cursor or Devin.
- Manual: set the JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables it shows, then follow Set up tracing.
Under Send sample traces, run your agent with 10 varied requests. Include successful ones and safe examples of failures.

The step completes once 10 distinct traces arrive.

Issues: Find recurring problems

Judgment starts a first detection run over the traces that arrive (how detection works).

Select Save & continue to keep a schedule, or No thanks to run detection only when you ask.

Judges: Track what matters in every trace

An agent also analyzes your traces and creates up to five judges with online monitoring on, so new traces start getting labels right away.

Slack: Keep your team in the loop

Select Set up Slack.
Connect your workspace. Only Admins and Owners can connect it.
Select the channels that get an issue report after each detection run. Developers, Admins, and Owners can select them.
Select Save & continue.

To skip, select Continue without setup, or Skip issue delivery if Slack is already connected. Your team then checks the Issues page instead.

Judgment Agent: Ask about anything in your project

The chat opens with a question already filled in: it asks the agent where your agent struggled and to cite the traces behind each pattern. Send it, then follow the citations back to the traces.

What you can do next

The cookbooks show best practices for using Judgment. They all build from the same Example agent.

Start with Issues, which reports recurring problems judges aren’t tracking yet. Root-cause an Issue takes one Issue from the inbox to a fix and a judge that watches for it.
To analyze your agent data, write a few judges. Labels as indexes explains how, and Data analysis walks through the setup.
To get alerted, add an action to a judge that sends an email or a Slack message when it applies a label. For cost and duration alerts, use a Condition judge with an action. Alerting builds both.
To test a change before you ship it, see Tests. Testing compares two versions of the example agent and gates pull requests in CI.
If you work in Codex, Claude Code, or Cursor, connect your coding agent over MCP. It can read your Issues, traces, and labels while it works, and fix an Issue you hand it.