Judgment Labs Logo
DocumentationSDK ReferenceChangelog
Judgeval Python-v1 SDKPrimitives

BaseCustomScorer

An abstract base class for creating custom evaluation scorers

An abstract base class for creating custom evaluation scorers. This generic class allows you to define specialized scoring logic for different data types.

Parameters

T

:TypeVar

Generic type parameter that defines the data type the scorer operates on

Example: Example, Trace

Methods

scorerequired

:def

Abstract method that produces an output score and reason for the given data. Must be implemented by subclasses.

score.py
def score(self, data: T) -> CustomScorerResult:
    # Custom scoring logic here
    return CustomScorerResult(score=1.0, reason="...")

Usage

BaseCustomScorer is not typically used directly. Instead, use one of its typed variants:

  • ExampleScorer: For scoring individual examples (BaseCustomScorer[Example])
  • TraceScorer: For scoring traces (BaseCustomScorer[Trace])

Trace

A list of trace spans representing the execution flow of your AI system

ExampleCustomScorer

A custom scorer class for creating specialized evaluation logic for examples

Last updated on

On this page

ParametersMethodsUsage