Judgment concepts

Learn how Judgment turns agent traces into production monitoring and offline tests.

Judgment turns production evidence into reusable evaluation for live monitoring and offline tests.

Shared foundations

Traces capture what happened

A trace records one agent run: model and tool calls, application logic, inputs, outputs, timing, cost, and errors. Monitoring and tests use that record as evidence.

For the sales assistant, one trace can begin with “Summarize my last meeting with Acme and draft a follow-up” and end with the completed draft. Meeting search, transcript retrieval, account lookup, and generation are child spans.

Judges produce evaluation results

A judge defines what to evaluate. Agent Judges use natural-language rubrics; Code Judges use Python. Each evaluation produces a result with a value, reasoning, and evidence.

The sales assistant might use judges for Request type, Follow-up grounding, and Meeting search failure. Together, they distinguish what the user wanted, whether the agent succeeded, and whether its environment worked.

Online monitoring

Behaviors make judge outputs monitorable

A behavior is a binary or categorical judge output selected for production monitoring. It belongs to its judge; numeric outputs remain scores.

Monitoring evaluates production traffic

Monitoring evaluates incoming traces with known behaviors, shows their rate and volume, and can trigger alerts. Production Issues surface recurring problems not already encoded by a judge.

Use a behavior when the criterion is already known, such as an unsupported follow-up. Use Production issues when you want Judgment to discover recurring patterns you have not defined.

Offline testing

Datasets preserve evidence

A dataset is a versioned set of examples, traces, or sessions kept for calibration and testing. Test runs stay pinned to the dataset version they evaluated.

Tests evaluate a fixed change

An offline test applies selected judge versions to one dataset version, keeping baseline and candidate comparisons reproducible as those assets change.

Save traces where the sales assistant used the wrong meeting, then run the current and proposed agent against the same cases. The comparison shows whether the change fixed the failure without changing the evidence.

Use the concepts in dependency order

Instrument the agent and verify that each trace contains the evidence a teammate would need to reconstruct the run.
Create judges for the requests, outcomes, or environment conditions you need to evaluate.
Link known binary or categorical outputs as behaviors for production monitoring.
Save representative production evidence in datasets and use it in offline tests before releasing a change.

Create judges before behaviors. Tests require both judges and a dataset.

Judgment Agent combines the current platform page with project tools to investigate traces, judges, behaviors, tests, and production issues. The Judgment MCP server provides project access to coding agents.

Next step

Use Judgment Agent to investigate the trace and judge result from the Quickstart.