--- title: "Create a judge" description: "Create a judge, choose how it evaluates and what it does with its labels, and publish new versions." sidebar: label: "Create a judge" seo: title: "Create a judge | Judgment Docs" description: "Create a judge, choose how it evaluates and what it does with its labels, and publish new versions." --- [Judges](/documentation/judges/overview) covers the evaluation methods. To have a judge label live traffic, see [Run a judge on live traffic](/documentation/judges/run-on-live-traffic). ## Let Judgment create the judge This is the default path, and usually the fastest. 1. Open **Judges** and select **New Judge**. 2. In **What do you want the judge to evaluate?**, describe the question in plain words. For the [lunch-order agent](/documentation/start/example-agent): `Flag traces where a restaurant's menu lookup failed and the agent dropped it without telling the user.` 3. Submit. This opens a Judgment Agent chat that drafts the rubric and output for that question and creates the judge. The dialog also offers two starting prompts. **Suggest judges to create** has the agent look through your traces and sessions and recommend judges, with the activity that motivated each one; it doesn't create anything until you pick. **Create a judge with a dataset** has it read the examples in a dataset you mention and propose a judge for you to confirm. The Judgment Agent asks before it writes anything unless you've set it to **Auto-allow writes**. See [Judgment Agent](/documentation/agents/judgment-agent). ## Create a judge manually 1. Open **Judges**, select **New Judge**, then choose **Create manually**. 2. Name the judge and, optionally, add a description that tells teammates when to use it. 3. Under **Evaluation method**, choose **Agentic Judge** or **Deterministic conditions**. - For **Agentic Judge**, write the rubric under **Instructions**, pick a **Reasoning level**, and define at least two **Choices**, each with a description. - For **Deterministic conditions**, add the conditions to check. The two choices default to **Matched** and **Not matched**. The conditions are described in [Judge actions](/documentation/judges/judge-actions#alert-on-cost-duration-or-errors). 4. Optionally set **Run settings** and **Actions** now, or add them later. 5. Select **Create Judge**. Creating the judge doesn't turn on online monitoring unless you turned it on under **Run settings**. Custom code judges can't be created here; see [Code judges](#code-judges). ## Write a rubric that works The judge only has the rubric to go on, so be specific: - Ask one question. "Was the request completed?" and "Was the user polite?" are two judges. - Say what counts as evidence. "The user asked for 30 people and the confirmed order covers fewer than 30" beats "the order was wrong." - Define each choice so two people would pick the same one. For order size: Small is under $200, Medium is $200 to $500, Large is over $500. - Name the edge cases. "If the agent asks a clarifying question and the user never answers, choose Not completed." To get help, open the **Details** tab and use **Improve with Agent**. Once the judge has labeled some traces, [align it](/documentation/judges/align-a-judge) against your own calls. ## Choose a scope and reasoning level - By default a judge evaluates individual traces. Turn on **Session scoring** under **Run settings** on the **Details** tab to evaluate the whole session instead, which fits questions like sentiment or "did the conversation resolve the request." Session scoring appears once **Online monitoring** is on; see [Run a judge on live traffic](/documentation/judges/run-on-live-traffic). Condition judges evaluate traces only. - **Reasoning level** is under **Evaluation method** on the **Details** tab. **Lite** (the default) is the fastest and lowest cost. **Normal** balances quality and cost. **High** is the most thorough and the most expensive. You don't pick a model; the level decides. Judge runs use Judgment Units, see [Billing and plans](/documentation/administration/billing-and-plans). ## Versions Changing the rubric, choices, reasoning level, run settings or actions creates a new version. When you select **Save**, choose a minor or major **New version**, then: - **Create draft** saves the change without affecting live traffic. - **Publish to production** makes it the version that online monitoring uses. To switch production to another version, pick it in the version menu and select **Promote to Production**. The badges **latest** and **production** show which is which. ## Actions Actions send an email, post to Slack, add the trace to a dataset, or run other judges when this judge returns a given result. They're organized in action groups under **Actions** on the **Details** tab; [Judge actions](/documentation/judges/judge-actions) covers the options. ## JEV Optimization JEV Optimization can make your Agent judges faster and cheaper. A preprocessing script extracts the information your judge needs from each trace, then JEV evaluates that evidence using your existing rubric. If the script can't extract the evidence, or JEV's result has low confidence, the ordinary Agent judge evaluates the trace instead. Ask the [Judgment Agent](/documentation/agents/judgment-agent) to turn on JEV Optimization for your judge. The agent inspects your traces, tests the extraction, and compares the results with the ordinary judge. Then it recommends whether JEV Optimization can be applied to this judge safely. You can review the results before deciding whether to enable it, and you can turn it off later. JEV Optimization is enabled per organization. If the Judgment Agent can't set it up for you, [contact support](/documentation/start/get-help). If you enable it on a judge: 1. Keep the trace structure consistent across the traces this judge evaluates, since the preprocessing script is built for that structure. 2. Update JEV Optimization if that structure changes. 3. Set up JEV Optimization again for each new judge version. A new version you save doesn't carry the setup over. JEV Optimization supports whole-trace Agent judges whose output is a set of choices, running on Judgment's hosted service with bring-your-own-keys (BYOK) turned off. It doesn't support numeric outputs, Code judges or session scoring. Exploration and setup use Judgment Units for model calls and backtesting, even if no suitable setup is enabled in the end. JEV doesn't explain its result. When JEV's result is accepted, the evaluation's reasoning shows JEV's confidence instead of an explanation. ## Create a judge from your coding agent With the [Judgment MCP server](/documentation/connect/connect-your-coding-agent) connected, ask your coding agent to create the judge, for example: ```text Use the Judgment MCP create_judge tool to create an Agent judge in lunch-order-agent named "Order size", with outputs Small (under $200), Medium ($200 to $500) and Large (over $500). The rubric should read the total from the estimate_order tool call. ``` `create_judge` takes `judgeType: "prompt"` for an Agent judge, `"deterministic"` for a Condition judge or `"custom"` for a Code judge, and an `initialVersion` with `scoreType: "categorical"` and at least two `outputs` (exactly two for a Condition judge, match first). See [MCP tools](/documentation/reference/mcp-tools). ## Code judges A Code judge is a Python class that subclasses `Judge` from the `judgeval` SDK, packaged as a `tar.gz` bundle. To create one, or upload a new version of one, use the bundle upload API or the MCP `create_judge` and `update_judge` tools. The app can't create Code judges, but it shows them and lets you change their run settings and actions. Enter keys your code reads at runtime in **Environment variables**, under **Evaluation method** on the judge's **Details** tab. The bundle format, upload request and limits are in [Code judge bundle](/documentation/reference/code-judge-bundle).