Skip to Content
Judgment Labs
Agent judges

AgentJudgeFactory

Create and update prompt-based Agent judges on the Judgment platform.

Prop

Type

Create and update prompt-based Agent judges on the Judgment platform.

Access this via client.agent_judges — you don't instantiate it directly.

client = Judgeval(project_name="my-project")

judge = client.agent_judges.create(
    name="helpfulness",
    prompt="Did the assistant answer the user's question?",
    score_type="binary",
)

client.agent_judges.update(
    judge_id=judge.judge_id,
    prompt="Updated rubric prompt.",
)

__init__()

def __init__(client, project_id, project_name):

Parameters

Prop

Type


create()

Create a new Agent judge.

Raises

ValueError: If score_type is "numeric".

def create(*, name, prompt, score_type, model=None, reasoning_level=None, description=None, judge_description=None, categories=None, min_score=None, max_score=None) -> typing.Optional:

Parameters

Prop

Type

Returns

typing.Optional - The created AgentJudge, or None if the project is unresolved.


update()

Update an existing Agent judge.

Passing any version field (prompt, reasoning_level, score_type, description, categories, min_score, max_score, or a source/target version) writes a new production version, copying unspecified fields from the source version. When target_major_version / target_minor_version are omitted, the latest version's minor is bumped by 1 — matching the UI's default "save" behaviour. The score type of a judge cannot change.

def update(*, judge_id, prompt=None, model=None, reasoning_level=None, score_type=None, description=None, judge_description=None, categories=None, min_score=None, max_score=None, source_major_version=None, source_minor_version=None, target_major_version=None, target_minor_version=None) -> typing.Optional:

Parameters

Prop

Type

Returns

typing.Optional - The updated AgentJudge, or None if the project is unresolved.


_resolve_target_version()

The version to write: latest minor + 1 in the latest major by default.

def _resolve_target_version(latest, major, minor) -> TargetVersion:

Parameters

Prop

Type

Returns

TargetVersion


_build_version_config()

The requested rubric fields, the rest copied from source.

def _build_version_config(source, *, prompt, description, reasoning_level, min_score, max_score) -> VersionConfig:

Parameters

Prop

Type

Returns

VersionConfig


_resolve_score_type()

The judge's own score type; binary also names create()'s Pass/Fail categorical judges.

def _resolve_score_type(requested, source) -> ScoreType:

Parameters

Prop

Type

Returns

ScoreType


_build_prompt_version()

A complete new version; monitoring settings and action groups come from source.

def _build_prompt_version(source, target, score_type, categories, config) -> PromptJudgeVersion:

Parameters

Prop

Type

Returns

PromptJudgeVersion


_build_category_outputs()

Existing outputs by ID (all of them when categories is None), new names by name and description.

def _build_category_outputs(source, categories) -> List[CategoryOutput]:

Parameters

Prop

Type

Returns

List[CategoryOutput]


_warn_if_model()

def _warn_if_model(model) -> None:

Parameters

Prop

Type

Returns

None


_to_agent_judge()

def _to_agent_judge(judge) -> AgentJudge:

Parameters

Prop

Type

Returns

AgentJudge

On this page