Dataset
Dataset class for managing datasets of Examples and Traces in Judgeval
The Dataset class lets you create, retrieve, and manage reusable evaluation datasets that are visible on the Judgment platform.
from judgeval import Judgeval
from judgeval.v1.data.example import Example
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.create(
name="qa_dataset",
examples=[Example.create(input="What is the powerhouse of the cell?", actual_output="The mitochondria.")]
)
dataset = judgeval.datasets.get(name="qa_dataset")
examples = []
example = Example.create(input="Sample question?", actual_output="Sample answer.")
examples.append(example)
dataset.add_examples(examples)judgeval.datasets.create()
Create a new evaluation dataset for storage and reuse across multiple evaluation runs.
judgeval.datasets.create(
name: str,
examples: Iterable[Example] = [],
overwrite: bool = False,
batch_size: int = 100
)Parameters
namerequired
:strexamples
:Iterable[Example]Iterable of examples to include in the dataset. See Example for details on the structure.
overwrite
:boolWhether to overwrite an existing dataset with the same name.
batch_size
:intNumber of examples to upload in each batch when creating the dataset.
Returns
A Dataset instance for further operations, or None if the project was not found during client initialization.
Example
from judgeval import Judgeval
from judgeval.v1.data.example import Example
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.create(
name="qa_dataset",
examples=[Example.create(input="What is the powerhouse of the cell?", actual_output="The mitochondria.")]
)Exceptions
JudgmentAPIError
:ExceptionRaised when a dataset with the same name already exists in the project and
overwrite=False. See
JudgmentAPIError
for details.
judgeval.datasets.get()
Retrieve a dataset from the Judgment platform by its name.
judgeval.datasets.get(
name: str
)Parameters
namerequired
:strReturns
A Dataset instance for further operations, or None if the project was not found during client initialization.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
if dataset:
print(dataset.examples)judgeval.datasets.list()
List all datasets in the project.
judgeval.datasets.list()Parameters
None
Returns
A list of DatasetInfo objects containing dataset metadata, or None if the project was not found during client initialization.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
datasets = judgeval.datasets.list()
for dataset_info in datasets:
print(f"{dataset_info.name}: {dataset_info.entries} examples")dataset.add_examples()
Add Examples to the dataset once you have created or retrieved it.
dataset.add_examples(
examples: Iterable[Example],
batch_size: int = 100
)Parameters
examplesrequired
:Iterable[Example]batch_size
:intReturns
None - examples are uploaded to the platform in batches with progress tracking.
Example
from judgeval import Judgeval
from judgeval.v1.data.example import Example
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
example = Example.create(input="Sample question?", actual_output="Sample answer.")
dataset.add_examples(examples=[example])dataset.add_from_json()
Load examples from a JSON file and add them to the dataset.
dataset.add_from_json(
file_path: str,
batch_size: int = 100
)Parameters
file_pathrequired
:strbatch_size
:intReturns
None - examples are loaded from the file and uploaded to the platform in batches with progress tracking.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
dataset.add_from_json("/path/to/examples.json")dataset.add_from_yaml()
Load examples from a YAML file and add them to the dataset.
dataset.add_from_yaml(
file_path: str,
batch_size: int = 100
)Parameters
file_pathrequired
:strbatch_size
:intReturns
None - examples are loaded from the file and uploaded to the platform in batches with progress tracking.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
dataset.add_from_yaml("/path/to/examples.yaml")dataset.save_as()
Export the dataset to a local file.
dataset.save_as(
file_type: Literal["json", "yaml"],
dir_path: str,
save_name: Optional[str] = None
)Parameters
file_typerequired
:Literal["json", "yaml"]dir_pathrequired
:strDirectory path where the file will be saved.
save_name
:Optional[str]Name for the saved file. If not provided, uses a timestamp.
Returns
None - the dataset is saved to a file in the specified directory.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
dataset.save_as("json", "/path/to/save/dir", "my_dataset")
dataset.save_as("yaml", "/path/to/save/dir", "my_dataset")dataset.display()
Display a formatted table of the dataset contents.
dataset.display(
max_examples: int = 5
)Parameters
max_examples
:intMaximum number of examples to display in the table.
Returns
None - displays the dataset information and examples in a formatted table.
Example
from judgeval import Judgeval
judgeval = Judgeval(project_name="default_project")
dataset = judgeval.datasets.get(name="qa_dataset")
dataset.display(max_examples=10)Return Types
Dataset
The Dataset object contains the following properties:
namereadonly
:strexamplesreadonly
:Optional[List[Example]]List of examples contained in the dataset. See Example for details on the structure.
dataset_kindreadonly
:strThe Dataset class also supports:
len(dataset)- Returns the number of examplesiter(dataset)- Iterate over examplesstr(dataset)- String representation
DatasetInfo
The DatasetInfo object contains the following properties:
dataset_idreadonly
:strnamereadonly
:strcreated_atreadonly
:strISO timestamp of when the dataset was created