Create a dataset
Create a dataset in the Judgment platform and define the schema every example must follow.
Create a dataset when you need a durable collection of examples, production traces, or sessions. At the end of this guide, the dataset exists with a schema that is ready for its first row.
Prerequisites
- A Judgment project where you can create datasets
- The fields each example needs
- A decision about whether tests will score saved traces, whole sessions, or fresh agent executions
Create and define the dataset
Create the dataset
Open Datasets and select New Dataset. Enter a unique name such as
Support regressions, then select Create.
The initial dialog creates an empty dataset. Schema definition happens on the dataset page.
Open the new dataset
Select the dataset from the Datasets table. The empty page prompts you to define the fields before adding examples.
Define the schema
Select Define schema. Start from a template or choose Custom:
- Trace only stores one saved trace per row.
- Session only stores one saved session per row.
- LLM Eval maps trace input and output into structured fields and keeps the trace link.
- Full Trace also maps trace metadata such as duration, cost, tags, and behaviors.
For a custom input set, add fields such as input and
expected_output. Use string, number, boolean, or json for literal
data. Add one trace or session field when platform-started tests should
score saved evidence.
A fresh-agent SDK test may start from literal fields alone because the SDK attaches a newly generated trace for each example.
Save the schema
Select Save schema. Check field names and types before adding the first example: once a dataset has ever contained an example, its schema is locked, even if every example is later removed.
Verify the dataset
Open Edit schema and confirm that every field and type matches the intended row shape. The dataset is ready when the schema is visible and the Examples tab shows no rows yet.
Troubleshooting
- If Define schema is unavailable, confirm that the dataset is not a read-only legacy dataset.
- If Edit schema has become View schema, the dataset has already held an example. Create a new dataset when the row contract must change.
- If a test config says that judges have nothing to evaluate, add a
traceorsessionfield before adding the first example, or run the agent through the SDK so every example receives a fresh trace. - If trace mapping reports a type mismatch, use a field type that matches the selected trace field or correct the schema before it locks.
Next step
Add production traces to the dataset, or use the dataset SDK reference to add structured examples programmatically.