Core Data Types

Overview

Core data types represent the fundamental objects you'll work with when using the JudgmentEval SDK. These types are used across multiple SDK components for evaluation, tracing, and dataset management.

`Example`

Represents a single evaluation example containing input data and expected outputs for testing AI systems.

Properties

`input`

str

Required

The input prompt or query to be evaluated

Example:

"What is the capital of France?"

`expected_output`

str

Optional

The expected or ideal response for comparison during evaluation

Example:

"The capital of France is Paris."

`actual_output`

str

Optional

The actual response generated by the system being evaluated

Example:

"Paris is the capital city of France."

`retrieval_context`

str

Optional

Additional context retrieved from external sources (e.g., RAG systems)

Example:

"According to Wikipedia: Paris is the capital and most populous city of France..."

`additional_metadata`

Dict[str, Any]

Optional

Extended metadata for storing custom fields and evaluation-specific information

Example:

{
    "model_version": "gpt-4-0125",
    "temperature": 0.7,
    "response_time_ms": 1250
}

`metadata`

Dict[str, Any]

Optional

Additional context or information about the example

Example:

{
    "category": "geography",
    "difficulty": "easy",
    "source": "world_facts_dataset"
}

Usage Examples

from judgeval.data import Example

# Basic example
example = Example(
    input="What is 2 + 2?",
    expected_output="4"
)

# Example with evaluation results
evaluated_example = Example(
    input="What is the capital of France?",
    expected_output="Paris",
    actual_output="Paris is the capital city of France.",
    metadata={
        "category": "geography",
        "difficulty": "easy"
    }
)

# RAG example with retrieval context
rag_example = Example(
    input="Explain quantum computing",
    expected_output="Quantum computing uses quantum mechanical phenomena...",
    actual_output="Quantum computing is a revolutionary technology...",
    retrieval_context="According to research papers: Quantum computing leverages quantum mechanics...",
    additional_metadata={
        "model_version": "gpt-4-0125",
        "temperature": 0.7,
        "retrieval_score": 0.95
    }
)

`ExampleScorer`

A custom scorer class that extends BaseScorer for creating specialized evaluation logic for individual examples.

Properties

`score_type`

str

Required

Type identifier for the scorer, defaults to "Custom"

Example:

"Custom"

`required_params`

List[str]

Optional

List of required parameters for the scorer

Example:

["temperature", "model_version"]

Methods

`a_score_example`

async def

Required

Asynchronously measures the score on a single example. Must be implemented by subclasses.

Example:

async def a_score_example(self, example: Example, *args, **kwargs) -> float:
    # Custom scoring logic here
    return score

Core Data Types

Properties

Usage Examples

Properties

Methods

On this page