---
title: "Datasets"
description: "Understand how datasets preserve reusable examples and production evidence for judge calibration and offline tests."
sidebar:
  label: "Datasets"
seo:
  title: "Datasets | Judgment Concepts"
  description: "Learn how Judgment datasets organize schema-enforced examples, trace-backed rows, and versioned test inputs."
---

A dataset is a reusable project asset that preserves the evidence you want to
evaluate again. Judges can use it for calibration, Tests can run over a fixed
version, and automations can add new production cases without moving the
dataset into any one workflow.

```mermaid
flowchart LR
  Trace["Production trace"] -->|"saved as"| Example["Example"]
  Input["Literal input"] -->|"stored as"| Example
  Example -->|"included in"| Version["Dataset version"]
  Version -->|"used by"| Test["Test run"]
```

## The dataset contract

Every dataset has one schema. The schema names each field and assigns it a
type: `string`, `number`, `boolean`, `json`, `trace`, or `session`. Every row is
an example and must contain every field declared by that schema.

The schema is editable until the first example is added. After that, it is
locked so every version and downstream test keeps the same row shape. Dataset
versions record membership changes; they do not create different schemas.

Common data fields include:

| Field | Purpose |
| --- | --- |
| `input` | The request or task supplied to an agent. |
| `actual_output` | The output captured from an existing execution. |
| `expected_output` | A reviewer-authored result or property the candidate output should satisfy. |
| Custom fields | Context, labels, metadata, or structured inputs needed by the agent or judges. |

These names are conventions, not reserved fields. Choose names that match the
agent inputs and judge context your team uses.

## Example-backed and trace-backed rows

All dataset rows are examples. What they can evaluate depends on the fields in
the row:

- An **example-backed row** stores literal inputs, expected outputs, labels, or
  other structured data. An SDK offline run can execute an agent with those
  fields and attach the fresh trace to the result.
- A **trace-backed row** also contains one field whose type is `trace`. Its
  value points to the saved production trace that judges should score. Adding a
  production trace through the platform creates this kind of row and copies
  the trace into offline storage.
- A **session-backed row** uses a `session` field instead when the evaluation
  target is a whole multi-trace session.

A dataset can contain at most one trace or session pointer. Platform-started
test runs need one of those pointers because the platform does not execute your
agent. SDK runs may instead provide a fresh agent trace for every example.

## Versions are stable membership snapshots

Adding or removing a batch of examples creates a new dataset version. Earlier
versions remain available, including rows removed from the latest version. A
test run resolves one version when it starts and stores that version with the
run, so later dataset edits cannot change what the run evaluated.

For the complete procedure, see [Manage dataset
versions](/documentation/datasets/versions).

## Where datasets fit

Datasets are shared assets between two parallel workflows:

- [Monitoring](/documentation/monitoring) finds production evidence worth
  preserving.
- [Tests](/documentation/tests/offline-tests) evaluates a fixed dataset with
  reusable judges before a change is released.

Test configurations reference [judges](/documentation/judges) and datasets.
They do not copy or own them.

## Next steps

- [Create a dataset](/documentation/datasets/creation) when you need a new
  schema-enforced collection.
- [Add production traces to a
  dataset](/documentation/datasets/production-traces) to preserve real
  failures and edge cases.
- [Run an offline test](/documentation/tests/offline-tests) after the dataset
  contains the evidence you want to evaluate.