Evaluation
EvaluatorRunner
Abstract base for evaluation runners.
Prop
Type
Abstract base for evaluation runners.
Concrete implementations handle either hosted (server-side) or local
(in-process) scorer execution. The generic parameter S is str
for hosted scorers or Judge for local scorers.
__init__()
def __init__(client, project_id, project_name):Parameters
Prop
Type
run()
Execute an evaluation run and return results.
def run(examples, scorers, eval_run_name, assert_test=False, timeout_seconds=300) -> typing.List:Parameters
Prop
Type
Returns
typing.List - A list of ScoringResult objects, one per example.
_binary_label()
def _binary_label(value) -> str:Parameters
Prop
Type
Returns
str
_scorer_value()
def _scorer_value(scorer_dict) -> str | float | None:Parameters
Prop
Type
Returns
str | float | None