Dataset
In most scenarios, you will have multiple examples that you want to evaluate together.
The EvalDataset class is used to manage collections of examples.
Datasets in judgeval allow you to scale evaluations and save, load, and synchronize datasets with the Judgment platform.
Creating a Dataset
Datasets can be created by passing a list of examples to the EvalDataset constructor.
from judgeval.data import Example
from judgeval.data.datasets import EvalDataset
examples = [
Example(input="...", actual_output="..."),
Example(input="...", actual_output="..."),
...
]
dataset = EvalDataset(
examples=examples
)You can also add Examples to an existing EvalDataset.
dataset.add_example(Example(input="Question 3?", actual_output="Answer 3."))Saving/Loading Datasets
You can save and load EvalDataset objects locally and interact with the Judgment Platform.
Remote access: Judgment Platform
Local Formats: JSON, CSV, YAML
Evaluate On Your Dataset / Examples
You can use the JudgmentClient to evaluate a collection of Examples using scorers.
from judgeval import JudgmentClient
from judgeval.scorers import FaithfulnessScorer
client = JudgmentClient()
res = client.run_evaluation(
examples=dataset.examples,
scorers=[FaithfulnessScorer(threshold=0.9)],
model="gpt-4.1",
)Exporting Datasets
You can export your datasets from the Judgment Platform UI for backup purposes or sharing with team members.
Export from Platform UI
- Navigate to your project in the Judgment Platform
- Select the dataset you want to export
- Click the "Download Dataset" button in the top right
- The dataset will be downloaded as a JSON file
The exported JSON file contains the complete dataset information, including metadata and examples:
{
"dataset_id": "f852eeee-87fa-4430-9571-5784e693326e",
"organization_id": "0fbb0aa8-a7b3-4108-b92a-cc6c6800d825",
"dataset_alias": "QA-Pairs",
"comments": null,
"source_file": null,
"created_at": "2025-04-23T22:38:11.709763+00:00",
"examples": [
{
"example_id": "119ee1f6-1046-41bc-bb89-d9fc704829dd",
"input": "How can I start meditating?",
"actual_output": null,
"expected_output": "Meditation is a wonderful way to relax and focus...",
"context": null,
"retrieval_context": null,
"additional_metadata": {
"synthetic": true
},
"tools_called": null,
"expected_tools": null,
"name": null,
"created_at": "2025-04-23T23:34:33.117479+00:00",
"dataset_id": "f852eeee-87fa-4430-9571-5784e693326e",
"eval_results_id": null,
},
// more examples...
]
}