Agent Judges

Use natural-language rubrics to evaluate traces and examples with structured results and supporting evidence.

An Agent Judge evaluates a trace or example against a natural-language rubric. It returns a structured value with reasoning and evidence from the evaluated input. Use one when the criterion requires interpretation rather than a fully deterministic rule.

Create an Agent Judge

This workflow creates a reusable rubric-based judge.

Prerequisites

A Judgment project where you can create judges

Choose the output type

Open Judges, select New Judge, then choose Create manually.

Select the output contract that matches the decision you need:

Binary for a true-or-false criterion
Classification for one choice from a defined set
Score for a numeric value within a range

This guide uses Binary.

Define the rubric

Name the judge Grounded answer and add a description that tells teammates when to use it. Keep the default reasoning level for the first run.

Enter this prompt:

Return true when the response is supported by information retrieved in the
trace. Return false when the response makes a claim that the retrieved
information does not support. Explain the evidence for the decision.

Select Create Judge. Judgment opens the new judge detail page.

The judge now exists as a reusable project asset. Creating it does not enable continuous online evaluation.

Create the same kind of judge from the SDK

Use the SDK when the judge definition should live beside application or infrastructure code.

from judgeval import Judgeval


client = Judgeval(project_name="my-project")

judge = client.agent_judges.create(
    name="Grounded answer",
    prompt=(
        "Return true when the response is supported by information "
        "retrieved in the trace. Explain the evidence."
    ),
    model="gpt-5.2",
    score_type="binary",
)

print(judge.judge_id)

import { Judgeval } from "judgeval";

const client = await Judgeval.create({ projectName: "my-project" });

const judge = await client.agentJudges.create({
  name: "Grounded answer",
  prompt:
    "Return true when the response is supported by information " +
    "retrieved in the trace. Explain the evidence.",
  model: "gpt-5.2",
  scoreType: "binary",
});

console.log(judge.judgeId);

After SDK creation, open Judges and select Grounded answer to review its configuration.

Next step

To run a binary or categorical judge on incoming traffic, create and monitor a behavior. To keep the judge on demand, apply it only from your agent or test workflow.