Skip to Content
Judgment Labs
External Judges

ExternalJudgeFactory

Create external judges and submit their results on the Judgment platform.

Access this via client.external_judges — you don't instantiate it directly.

client = Judgeval(project_name="my-project")

judge = client.external_judges.create(
    name="human-thumbs-up",
    score_type="binary",
)

client.external_judges.submit_result(
    judge_id=judge.judge_id,
    target={"type": "trace", "trace_id": "<trace_id>"},
    value=True,
)

__init__()

def __init__(client, project_id, project_name):

Parameters

Prop

Type


create()

Create a new external judge and its first version.

Raises

ValueError: If outputs is missing/too short for a categorical judge, or provided for a non-categorical one.

JudgmentConflictError: If a judge with this name already exists in the project.

JudgmentValidationError: If the server rejects the judge configuration.

judge = client.external_judges.create(
    name="topic-classifier",
    score_type="categorical",
    outputs=[
        {"name": "billing", "description": "Billing questions"},
        {"name": "support", "description": "Support requests"},
    ],
)
def create(*, name, score_type, judge_description=None, outputs=None) -> typing.Optional:

Parameters

Prop

Type

Returns

typing.Optional - The created ExternalJudge, or None if the project is unresolved.


submit_result()

Submit a customer score under any judge for a trace, session, or conversation.

Resubmitting for the same judge and target replaces its current customer score. To correct a value, submit again for that judge and target. Each submission, including a retry, writes a new result row.

Exactly one of judge_id or judge_name must be provided; the score is attached to that judge's production version. value must match the judge's score type (bool for binary, int/ float for numeric, or one of the judge's configured outputs' names for categorical).

Raises

ValueError: If neither or both of judge_id/judge_name are provided, or the target is invalid.

JudgmentValidationError: If the server rejects the value or target.

client.external_judges.submit_result(
    judge_name="human-thumbs-up",
    target={"type": "trace", "trace_id": "<trace_id>"},
    value=True,
    reason="Reviewer approved the response.",
)
def submit_result(*, target, value, judge_id=None, judge_name=None, reason=None) -> typing.Optional:

Parameters

Prop

Type

Returns

typing.Optional - The id of the persisted score result, or None if the project is unresolved.

On this page