Dataset
Dataset class for managing datasets of Examples and Traces in Judgeval
Overview
The Dataset class provides both methods for dataset operations and serves as the return type for dataset instances. When you call Dataset.create() or Dataset.get(), you receive a Dataset instance with additional methods for managing the dataset's contents.
Quick Start Example
from judgeval.dataset import Dataset
from judgeval.data import Example
dataset = Dataset.create(
name="qa_dataset",
project_name="default_project",
examples=[Example(input="What is the powerhouse of the cell?", actual_output="The mitochondria.")]
)
dataset = Dataset.get(
name="qa_dataset",
project_name="default_project",
)
examples = []
example = Example(input="Sample question?", output="Sample answer.")
examples.append(example)
dataset.add_examples(examples=examples)Dataset Creation & Retrieval
Dataset.create()
Create a new evaluation dataset for storage and reuse across multiple evaluation runs. Note this command pushes the dataset to the Judgment platform.
Parameters
examples
Optional[List[[Example]]
Optional
List of examples to include in the dataset. See Example for details on the structure.
[Example(input="...", actual_output="...")]
traces
Optional[List[Trace]]
Optional
List of traces to include in the dataset. See Trace for details on the structure.
[Trace(...)]
overwrite
bool
Default: False
Optional
Whether to overwrite an existing dataset with the same name.
Returns
Returns
Dataset
Optional
A Dataset instance for further operations
Exceptions
JudgmentAPIError
exception
Optional
Raised when a dataset with the same name already exists in the project and overwrite=False. See JudgmentAPIError for details.
Example Code
from judgeval.dataset import Dataset
from judgeval.data import Example
dataset = Dataset.create(
name="qa_dataset",
project_name="default_project",
examples=[Example(input="What is the powerhouse of the cell?", actual_output="The mitochondria.")]
)Dataset.get()
Retrieve a dataset from the Judgment platform by its name and project name.
Parameters
project_name
str
Required
The name of the project where the dataset is stored.
"default_project"
Returns
Returns
Dataset
Optional
A Dataset instance for further operations
Example Code
from judgeval.dataset import Dataset
dataset = Dataset.get(
name="qa_dataset",
project_name="default_project",
)
print(dataset.examples)Dataset Management
Once you have a Dataset instance (from Dataset.create() or Dataset.get()), you can use these methods to manage its contents:
Note: All instance methods automatically update the dataset and push changes to the Judgment platform.
dataset.add_examples()
Add examples to the dataset.
Parameters
examples
List[Example]
Required
List of examples to add to the dataset.
Returns
Returns
bool
Optional
True if examples were added successfully
Example Code
from judgeval.dataset import Dataset
from judgeval.data import Example
dataset = Dataset.get(
name="qa_dataset",
project_name="default_project",
)
example = Example(input="Sample question?", output="Sample answer.")
examples = [example]
dataset.add_examples(examples=examples)Dataset Properties
When you have a Dataset instance, it provides access to the following properties:
Dataset
dataset.name
Type: str (read-only)
The name of the dataset.
dataset.project_name
Type: str (read-only)
The project name where the dataset is stored.
dataset.examples
Type: List[Example] (read-only)
List of examples contained in the dataset.
dataset.traces
Type: List[Trace] (read-only)
List of traces contained in the dataset (if any).
dataset.id
Type: str (read-only)
Unique identifier for the dataset on the Judgment platform.