ExternalJudgeFactory
Create external judges and submit their results on the Judgment platform.
Access this via client.external_judges — you don't instantiate it
directly.
client = Judgeval(project_name="my-project")
judge = client.external_judges.create(
name="human-thumbs-up",
score_type="binary",
)
client.external_judges.submit_result(
judge_id=judge.judge_id,
target={"type": "trace", "trace_id": "<trace_id>"},
value=True,
)__init__()
def __init__(client, project_id, project_name):Parameters
Prop
Type
create()
Create a new external judge and its first version.
Raises
ValueError: If outputs is missing/too short for a
categorical judge, or provided for a non-categorical one.
JudgmentConflictError: If a judge with this name already exists in the project.
JudgmentValidationError: If the server rejects the judge configuration.
judge = client.external_judges.create(
name="topic-classifier",
score_type="categorical",
outputs=[
{"name": "billing", "description": "Billing questions"},
{"name": "support", "description": "Support requests"},
],
)def create(*, name, score_type, judge_description=None, outputs=None) -> typing.Optional:Parameters
Prop
Type
Returns
typing.Optional - The created ExternalJudge, or None if the project is
unresolved.
submit_result()
Submit a customer score under any judge for a trace, session, or conversation.
Resubmitting for the same judge and target replaces its current customer score. To correct a value, submit again for that judge and target. Each submission, including a retry, writes a new result row.
Exactly one of judge_id or judge_name must be provided; the
score is attached to that judge's production version. value
must match the judge's score type (bool for binary, int/
float for numeric, or one of the judge's configured outputs'
names for categorical).
Raises
ValueError: If neither or both of judge_id/judge_name are
provided, or the target is invalid.
JudgmentValidationError: If the server rejects the value or target.
client.external_judges.submit_result(
judge_name="human-thumbs-up",
target={"type": "trace", "trace_id": "<trace_id>"},
value=True,
reason="Reviewer approved the response.",
)def submit_result(*, target, value, judge_id=None, judge_name=None, reason=None) -> typing.Optional:Parameters
Prop
Type
Returns
typing.Optional - The id of the persisted score result, or None if the
project is unresolved.