Skip to Content
Judgment Labs
Evaluation

EvaluatorRunner

Abstract base for evaluation runners.

Prop

Type

Abstract base for evaluation runners.

Concrete implementations handle either hosted (server-side) or local (in-process) scorer execution. The generic parameter S is str for hosted scorers or Judge for local scorers.

__init__()

def __init__(client, project_id, project_name):

Parameters

Prop

Type


run()

Execute an evaluation run and return results.

def run(examples, scorers, eval_run_name, assert_test=False, timeout_seconds=300) -> typing.List:

Parameters

Prop

Type

Returns

typing.List - A list of ScoringResult objects, one per example.


_binary_label()

def _binary_label(value) -> str:

Parameters

Prop

Type

Returns

str


_scorer_value()

def _scorer_value(scorer_dict) -> str | float | None:

Parameters

Prop

Type

Returns

str | float | None

On this page