BaseScorer

For reference only - users should create scorers via ExampleScorer instead of implementing BaseScorer directly

Abstract base class for implementing custom scoring logic.

scorerequired:Callable[[str, str, Optional[str]], float]

Main evaluation method that must be implemented by subclasses. Returns a numeric score for the given input/output pair.

Example:

score.py

def score(self, input: str, output: str, expected: str = None) -> float:
  # Custom scoring logic here
  return 0.85

get_name:Callable[[], str]

Returns the name/identifier for this scorer. Override to customize.

Reference Example

# BaseScorer is the abstract base class - for reference only
# In practice, create scorers using ExampleScorer:

from judgeval import ExampleScorer

# Create a custom scorer using ExampleScorer (recommended approach)
custom_scorer = ExampleScorer(
    name="similarity_scorer",
    scorer_fn=lambda input, output, expected: 1.0 if expected and expected.lower() in output.lower() else 0.0
)

# Use the scorer
result = custom_scorer.score(
    input="What is 2+2?",
    output="The answer is 4",
    expected="4"
)

BaseScorer

Reference Example

On this page