Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Create a judge

Create a judge, choose how it evaluates and what it does with its labels, and publish new versions.

Judges covers the evaluation methods. To have a judge label live traffic, see Run a judge on live traffic.

Let Judgment create the judge

This is the default path, and usually the fastest.

  1. Open Judges and select New Judge.
  2. In What do you want the judge to evaluate?, describe the question in plain words. For the lunch-order agent: Flag traces where a restaurant's menu lookup failed and the agent dropped it without telling the user.
  3. Submit. This opens a Judgment Agent chat that drafts the rubric and output for that question and creates the judge.

The dialog also offers two starting prompts. Suggest judges to create has the agent look through your traces and sessions and recommend judges, with the activity that motivated each one; it doesn’t create anything until you pick. Create a judge with a dataset has it read the examples in a dataset you mention and propose a judge for you to confirm.

The Judgment Agent asks before it writes anything unless you’ve set it to Auto-allow writes. See Judgment Agent.

Create a judge manually

  1. Open Judges, select New Judge, then choose Create manually.
  2. Name the judge and, optionally, add a description that tells teammates when to use it.
  3. Under Evaluation method, choose Agentic Judge or Deterministic conditions.
    • For Agentic Judge, write the rubric under Instructions, pick a Reasoning level, and define at least two Choices, each with a description.
    • For Deterministic conditions, add the conditions to check. The two choices default to Matched and Not matched. The conditions are described in Judge actions.
  4. Optionally set Run settings and Actions now, or add them later.
  5. Select Create Judge.

Creating the judge doesn’t turn on online monitoring unless you turned it on under Run settings. Custom code judges can’t be created here; see Code judges.

Write a rubric that works

The judge only has the rubric to go on, so be specific:

  • Ask one question. “Was the request completed?” and “Was the user polite?” are two judges.
  • Say what counts as evidence. “The user asked for 30 people and the confirmed order covers fewer than 30” beats “the order was wrong.”
  • Define each choice so two people would pick the same one. For order size: Small is under $200, Medium is $200 to $500, Large is over $500.
  • Name the edge cases. “If the agent asks a clarifying question and the user never answers, choose Not completed.”

To get help, open the Details tab and use Improve with Agent. Once the judge has labeled some traces, align it against your own calls.

Choose a scope and reasoning level

  • By default a judge evaluates individual traces. Turn on Session scoring under Run settings on the Details tab to evaluate the whole session instead, which fits questions like sentiment or “did the conversation resolve the request.” Session scoring appears once Online monitoring is on; see Run a judge on live traffic. Condition judges evaluate traces only.
  • Reasoning level is under Evaluation method on the Details tab. Lite (the default) is the fastest and lowest cost. Normal balances quality and cost. High is the most thorough and the most expensive. You don’t pick a model; the level decides. Judge runs use Judgment Units, see Billing and plans.

Versions

Changing the rubric, choices, reasoning level, run settings or actions creates a new version. When you select Save, choose a minor or major New version, then:

  • Create draft saves the change without affecting live traffic.
  • Publish to production makes it the version that online monitoring uses.

To switch production to another version, pick it in the version menu and select Promote to Production. The badges latest and production show which is which.

Actions

Actions send an email, post to Slack, add the trace to a dataset, or run other judges when this judge returns a given result. They’re organized in action groups under Actions on the Details tab; Judge actions covers the options.

JEV Optimization

JEV Optimization can make your Agent judges faster and cheaper. A preprocessing script extracts the information your judge needs from each trace, then JEV evaluates that evidence using your existing rubric. If the script can’t extract the evidence, or JEV’s result has low confidence, the ordinary Agent judge evaluates the trace instead.

Ask the Judgment Agent to turn on JEV Optimization for your judge. The agent inspects your traces, tests the extraction, and compares the results with the ordinary judge. Then it recommends whether JEV Optimization can be applied to this judge safely.

You can review the results before deciding whether to enable it, and you can turn it off later. JEV Optimization is enabled per organization. If the Judgment Agent can’t set it up for you, contact support.

If you enable it on a judge:

  1. Keep the trace structure consistent across the traces this judge evaluates, since the preprocessing script is built for that structure.
  2. Update JEV Optimization if that structure changes.
  3. Set up JEV Optimization again for each new judge version. A new version you save doesn’t carry the setup over.

JEV Optimization supports whole-trace Agent judges whose output is a set of choices, running on Judgment’s hosted service with bring-your-own-keys (BYOK) turned off. It doesn’t support numeric outputs, Code judges or session scoring.

Exploration and setup use Judgment Units for model calls and backtesting, even if no suitable setup is enabled in the end.

JEV doesn’t explain its result. When JEV’s result is accepted, the evaluation’s reasoning shows JEV’s confidence instead of an explanation.

Create a judge from your coding agent

With the Judgment MCP server connected, ask your coding agent to create the judge, for example:

Use the Judgment MCP create_judge tool to create an Agent judge in lunch-order-agent
named "Order size", with outputs Small (under $200), Medium ($200 to $500)
and Large (over $500). The rubric should read the total from the estimate_order tool call.

create_judge takes judgeType: "prompt" for an Agent judge, "deterministic" for a Condition judge or "custom" for a Code judge, and an initialVersion with scoreType: "categorical" and at least two outputs (exactly two for a Condition judge, match first). See MCP tools.

Code judges

A Code judge is a Python class that subclasses Judge from the judgeval SDK, packaged as a tar.gz bundle. To create one, or upload a new version of one, use the bundle upload API or the MCP create_judge and update_judge tools.

The app can’t create Code judges, but it shows them and lets you change their run settings and actions. Enter keys your code reads at runtime in Environment variables, under Evaluation method on the judge’s Details tab.

The bundle format, upload request and limits are in Code judge bundle.

Was this page helpful?