---
title: "Build a dataset"
description: "Create the test cases your agent runs on, in the app, from existing traces, or from code."
sidebar:
  label: "Build a dataset"
seo:
  title: "Build a dataset | Judgment Docs"
  description: "Create the test cases your agent runs on, in the app, from existing traces, or from code."
---

For the [lunch-order agent](/documentation/start/example-agent), a dataset is ten or so real requests: "order vegetarian lunch for 12 under $200", "order mediterranean for 30", "I need halal catering for 20 by 11:30".

Datasets live under **Datasets & Tests** in the app. You use them to run [test configs](/documentation/tests/run-a-test-from-your-code), to [label a dataset](/documentation/tests/label-a-dataset) against a judge, and to keep traces you want to look at again. [Test suites](/documentation/tests/overview) keep their own cases and don't read from datasets.

## Pick the fields first

Every dataset has a schema, and every test case must fill every field. Choose the fields by what you plan to do with the dataset:

- **To run your agent** on each case in a test config run, give the dataset the fields your agent takes as input. For the lunch-order agent that's one string field, `input`, holding the user's request. You can add extra fields, such as `budget`, for your pass condition to read. Don't pass reference answers to the agent as input.
- **To re-score stored traces** with a changed judge, or to [label the dataset](/documentation/tests/label-a-dataset), the dataset needs a trace field (or a session field) that points at existing traces.

Once a test case is added, the schema can't change.

## Ask the agent to build it

1. Open **Datasets & Tests** and select **New Dataset**.
2. The dialog asks "What test cases should we create for your agent?" Describe what you want, or pick a suggestion: **Turn recent failures into test cases** or **Build a regression set for my agent**.
3. Type `@` to add context, for example an Issue, a saved view, or a judge label. You can also attach files.
4. Send the prompt. The agent reads your project's traces and creates the dataset.

This is the fastest way to turn an [Issue](/documentation/issues/triage-and-fix) into test cases. Mention the Issue, for example "Menu lookups time out and the agent silently drops the restaurant", and ask for test cases that cover it.

## Add traces you already have

When you're looking at a trace that shows the problem, add it straight to a dataset:

- In the traces table, select one or more traces and choose **Add to Dataset**.
- In a single trace's view, use **Add to Dataset** from the header.
- Sessions have the same action.
- A [judge action](/documentation/judges/judge-actions) can add matching traces for you with **Add to dataset**.

In the dialog, pick an existing dataset or **Create New Dataset**. The schema presets decide what gets copied from the trace:

| Preset | Fields |
|---|---|
| Trace only | `trace` |
| Session only | `session` |
| LLM Eval | `input`, `actual_output`, `trace` |
| Full Trace | `input`, `output`, `span_name`, `duration`, `llm_cost`, `tags`, `behaviors` (the trace's labels), `trace` |

Field and API names still say behavior; it means label.

Issue evidence works the same way: open an evidence trace from the Issue and add it.

## Create one by hand

1. Select **New Dataset**, then **Create manually**.
2. In **Create empty dataset**, enter a name.
3. Select **Define schema** and add your fields, or start from a preset.
4. Fill it with **Add to Dataset** from traces, or from code (below). The app has no form for typing test cases in one by one, and no file upload.

## Create one from code

Code is the easiest way to keep a hand-written list of prompts in your repo and push it to Judgment.

**Python**

```python
from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="lunch-order-agent")

prompts = [
    "order vegetarian lunch for 12 under $200",
    "order mediterranean for 30",
    "I need halal catering for 20 by 11:30",
]

client.datasets.create(
    name="lunch-regression",
    schema={"type": "object", "properties": {"input": {"type": "string"}}},
    examples=[Example.create(input=p) for p in prompts],
)
```

**TypeScript**

```typescript
import { Example, Judgeval } from "judgeval";

const client = await Judgeval.create({ projectName: "lunch-order-agent" });

const prompts = [
  "order vegetarian lunch for 12 under $200",
  "order mediterranean for 30",
  "I need halal catering for 20 by 11:30",
];

await client.datasets.create("lunch-regression", {
  schema: { type: "object", properties: { input: { type: "string" } } },
  examples: prompts.map((input) => Example.create({ input })),
});
```

Both read `JUDGMENT_API_KEY` and `JUDGMENT_ORG_ID` from the environment. Dataset names are unique per project; `create` fails if the name exists unless you pass `overwrite=True` (Python) or `overwrite: true` (TypeScript), and overwrite is refused once test configs use the dataset. To append later in Python, fetch the dataset with `client.datasets.get(name="lunch-regression")` and call `add_examples(...)` or `add_from_json(path)`.

Coding agents can do the same through MCP with `create_dataset` and `add_examples_to_dataset`. See [MCP tools](/documentation/reference/mcp-tools).

## Versions

Every change to a dataset's test cases creates a new version. Adding cases creates one; removing cases creates one too, and earlier versions keep the removed cases. The dataset header shows **Version \{n\}**, and you can switch to an earlier version there.

A test config run uses the latest version unless you pin one with `dataset_version`. Pin it when you compare runs, so both runs grade the same test cases. If you pick a version in the header before starting a new run, the app's code template includes it for you.

To take test cases out of Judgment, export them from the dataset (the **Export Examples** dialog).