Build a dataset

Create the test cases your agent runs on, in the app, from existing traces, or from code.

For the lunch-order agent, a dataset is ten or so real requests: “order vegetarian lunch for 12 under $200”, “order mediterranean for 30”, “I need halal catering for 20 by 11:30”.

Datasets live under Datasets & Tests in the app. You use them to run test configs, to label a dataset against a judge, and to keep traces you want to look at again. Test suites keep their own cases and don’t read from datasets.

Pick the fields first

Every dataset has a schema, and every test case must fill every field. Choose the fields by what you plan to do with the dataset:

• To run your agent on each case in a test config run, give the dataset the fields your agent takes as input. For the lunch-order agent that’s one string field, input, holding the user’s request. You can add extra fields, such as budget, for your pass condition to read. Don’t pass reference answers to the agent as input.
• To re-score stored traces with a changed judge, or to label the dataset, the dataset needs a trace field (or a session field) that points at existing traces.

Once a test case is added, the schema can’t change.

Ask the agent to build it

1. Open Datasets & Tests and select New Dataset.
2. The dialog asks “What test cases should we create for your agent?” Describe what you want, or pick a suggestion: Turn recent failures into test cases or Build a regression set for my agent.
3. Type @ to add context, for example an Issue, a saved view, or a judge label. You can also attach files.
4. Send the prompt. The agent reads your project’s traces and creates the dataset.

This is the fastest way to turn an Issue into test cases. Mention the Issue, for example “Menu lookups time out and the agent silently drops the restaurant”, and ask for test cases that cover it.

Add traces you already have

When you’re looking at a trace that shows the problem, add it straight to a dataset:

• In the traces table, select one or more traces and choose Add to Dataset.
• In a single trace’s view, use Add to Dataset from the header.
• Sessions have the same action.
• A judge action can add matching traces for you with Add to dataset.

In the dialog, pick an existing dataset or Create New Dataset. The schema presets decide what gets copied from the trace:

Field and API names still say behavior; it means label.

Issue evidence works the same way: open an evidence trace from the Issue and add it.

Create one by hand

1. Select New Dataset, then Create manually.
2. In Create empty dataset, enter a name.
3. Select Define schema and add your fields, or start from a preset.
4. Fill it with Add to Dataset from traces, or from code (below). The app has no form for typing test cases in one by one, and no file upload.

Create one from code

Code is the easiest way to keep a hand-written list of prompts in your repo and push it to Judgment.

from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="lunch-order-agent")

prompts = [
    "order vegetarian lunch for 12 under $200",
    "order mediterranean for 30",
    "I need halal catering for 20 by 11:30",
]

client.datasets.create(
    name="lunch-regression",
    schema={"type": "object", "properties": {"input": {"type": "string"}}},
    examples=[Example.create(input=p) for p in prompts],
)

import { Example, Judgeval } from "judgeval";

const client = await Judgeval.create({ projectName: "lunch-order-agent" });

const prompts = [
  "order vegetarian lunch for 12 under $200",
  "order mediterranean for 30",
  "I need halal catering for 20 by 11:30",
];

await client.datasets.create("lunch-regression", {
  schema: { type: "object", properties: { input: { type: "string" } } },
  examples: prompts.map((input) => Example.create({ input })),
});

Both read JUDGMENT_API_KEY and JUDGMENT_ORG_ID from the environment. Dataset names are unique per project; create fails if the name exists unless you pass overwrite=True (Python) or overwrite: true (TypeScript), and overwrite is refused once test configs use the dataset. To append later in Python, fetch the dataset with client.datasets.get(name="lunch-regression") and call add_examples(...) or add_from_json(path).

Coding agents can do the same through MCP with create_dataset and add_examples_to_dataset. See MCP tools.

Versions

Every change to a dataset’s test cases creates a new version. Adding cases creates one; removing cases creates one too, and earlier versions keep the removed cases. The dataset header shows Version {n}, and you can switch to an earlier version there.

A test config run uses the latest version unless you pin one with dataset_version. Pin it when you compare runs, so both runs grade the same test cases. If you pick a version in the header before starting a new run, the app’s code template includes it for you.

To take test cases out of Judgment, export them from the dataset (the Export Examples dialog).