Configuration Types

Overview

Configuration types define how different components of the JudgmentEval SDK should behave. These types are used to customize scoring behavior, API clients, and evaluation parameters.

Internal Configuration Types

For reference only - users should create scorers via ExampleScorer instead of implementing BaseScorer or APIScorerConfig directly

`BaseScorer`

Abstract base class for implementing custom scoring logic.

Methods

`score(input: str, output: str, expected: str = None) -> float`

method

Required

Main evaluation method that must be implemented by subclasses. Returns a numeric score for the given input/output pair.

Example:

def score(self, input: str, output: str, expected: str = None) -> float:
    # Custom scoring logic here
    return 0.85

`get_name() -> str`

method

Optional

Returns the name/identifier for this scorer. Override to customize.

Reference Example

# BaseScorer is the abstract base class - for reference only
# In practice, create scorers using ExampleScorer:

from judgeval import ExampleScorer

# Create a custom scorer using ExampleScorer (recommended approach)
custom_scorer = ExampleScorer(
    name="similarity_scorer",
    scorer_fn=lambda input, output, expected: 1.0 if expected and expected.lower() in output.lower() else 0.0
)

# Use the scorer
result = custom_scorer.score(
    input="What is 2+2?",
    output="The answer is 4", 
    expected="4"
)

`APIScorerConfig`

Configuration object for built-in Judgment scorers.

Properties

`name`

str

Required

Unique identifier for the scorer configuration

Example:

"accuracy_scorer"

`prompt`

str

Required

The evaluation prompt that will be used to judge responses

Example:

"Rate the accuracy of this answer on a scale of 1-5, where 5 is completely accurate."

`options`

Dict[str, Any]

Optional

Additional configuration options for the scorer

Example:

{
    "model": "gpt-4",
    "temperature": 0.0,
    "max_tokens": 100
}

`judgment_api_key`

str

Optional

API key for Judgment platform authentication. Defaults to JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Organization identifier for API requests. Defaults to JUDGMENT_ORG_ID environment variable

Utility Types

Common Configuration Patterns

Type Aliases

`ScorerType`

Union[APIScorerConfig, BaseScorer]

Optional

Commonly used union type accepting either API configuration or custom scorer instances

`ConfigDict`

Dict[str, Any]

Optional

General-purpose configuration dictionary for flexible parameter passing

`OptionalConfig`

Optional[Dict[str, Any]]

Optional

Optional configuration dictionary, commonly used for metadata and additional options

File Format Types

`FileFormat`

Literal['json', 'yaml']

Optional

Supported file formats for dataset import/export operations

Example:

# Used in dataset export methods
dataset.save(
    file_type="json",  # or "yaml"
    dir_path="/path/to/save"
)

Configuration Types

Methods

Reference Example

Properties

Type Aliases

File Format Types

On this page