Datasets
Understand how datasets preserve reusable examples and production evidence for judge calibration and offline tests.
A dataset is a reusable project asset that preserves the evidence you want to evaluate again. Judges can use it for calibration, Tests can run over a fixed version, and automations can add new production cases without moving the dataset into any one workflow.
The dataset contract
Every dataset has one schema. The schema names each field and assigns it a
type: string, number, boolean, json, trace, or session. Every row is
an example and must contain every field declared by that schema.
The schema is editable until the first example is added. After that, it is locked so every version and downstream test keeps the same row shape. Dataset versions record membership changes; they do not create different schemas.
Common data fields include:
| Field | Purpose |
|---|---|
input |
The request or task supplied to an agent. |
actual_output |
The output captured from an existing execution. |
expected_output |
A reviewer-authored result or property the candidate output should satisfy. |
| Custom fields | Context, labels, metadata, or structured inputs needed by the agent or judges. |
These names are conventions, not reserved fields. Choose names that match the agent inputs and judge context your team uses.
Example-backed and trace-backed rows
All dataset rows are examples. What they can evaluate depends on the fields in the row:
- An example-backed row stores literal inputs, expected outputs, labels, or other structured data. An SDK offline run can execute an agent with those fields and attach the fresh trace to the result.
- A trace-backed row also contains one field whose type is
trace. Its value points to the saved production trace that judges should score. Adding a production trace through the platform creates this kind of row and copies the trace into offline storage. - A session-backed row uses a
sessionfield instead when the evaluation target is a whole multi-trace session.
A dataset can contain at most one trace or session pointer. Platform-started test runs need one of those pointers because the platform does not execute your agent. SDK runs may instead provide a fresh agent trace for every example.
Versions are stable membership snapshots
Adding or removing a batch of examples creates a new dataset version. Earlier versions remain available, including rows removed from the latest version. A test run resolves one version when it starts and stores that version with the run, so later dataset edits cannot change what the run evaluated.
For the complete procedure, see Manage dataset versions.
Where datasets fit
Datasets are shared assets between two parallel workflows:
- Monitoring finds production evidence worth preserving.
- Tests evaluates a fixed dataset with reusable judges before a change is released.
Test configurations reference judges and datasets. They do not copy or own them.
Next steps
- Create a dataset when you need a new schema-enforced collection.
- Add production traces to a dataset to preserve real failures and edge cases.
- Run an offline test after the dataset contains the evidence you want to evaluate.