Skip to Content
Judgment Labs
Judges

ExternalJudgeFactory

Create external judges and submit their results on the Judgment platform.

Access via client.externalJudges.

const judge = await client.externalJudges.create({
  name: "human-thumbs-up",
  scoreType: "binary",
});

await client.externalJudges.submitResult({
  judgeId: judge.judgeId,
  target: { type: "trace", traceId: "<trace_id>" },
  value: true,
});

create()

Create a new external judge and its first version.

const judge = await client.externalJudges.create({
  name: "topic-classifier",
  scoreType: "categorical",
  outputs: [
    { name: "billing", description: "Billing questions" },
    { name: "support", description: "Support requests" },
  ],
});
async function create(options: { name: string; scoreType: ScoreType; judgeDescription?: string; outputs?: { name: string; description: string; }[]; }): Promise<ExternalJudge | null>

Parameters

Prop

Type

Returns

Promise<ExternalJudge | null> - The newly created ExternalJudge, or null if the project is unresolved.


submitResult()

Submits a customer score under any judge for a trace, session, or conversation.

Resubmitting for the same judge and target replaces its current customer score. To correct a value, submit again for that judge and target. Each submission, including a retry, writes a new result row.

Exactly one of judgeId or judgeName must be provided; the score is attached to that judge's production version. value must match the judge's score type (boolean for binary, number for numeric, or one of the judge's configured outputs' names for categorical).

await client.externalJudges.submitResult({
  judgeName: "human-thumbs-up",
  target: { type: "trace", traceId: "<trace_id>" },
  value: true,
  reason: "Reviewer approved the response.",
});
async function submitResult(options: { target, value, judgeId?, judgeName?, reason? }): Promise<string | null>

Parameters

Prop

Type

Returns

Promise<string | null> - The id of the persisted score result, or null if the project is unresolved.

On this page