Label a dataset

Review trace or session examples against a judge and create a separate dataset with human-expected labels.

Use Label on a dataset to record the outputs a judge should return for the examples in it. Finishing creates a new dataset with the human labels; it leaves the source dataset unchanged.

Prerequisites

• A dataset with a trace or session field. See Build a dataset.
• A judge with a production version that matches the dataset: a trace judge for traces, or a judge with session scoring for sessions.
• Permission to create a dataset.

If you already reviewed judge results on a judge’s Alignment tab, you can work from those reviews instead; see Align a judge.

Record the expected outputs

1. Open Datasets & Tests, select the source dataset, and pick the version in the header.
2. Select Label in the header.
3. In Label Dataset, choose a Judge, enter the New dataset name, and select Start labeling.
4. Inspect each example’s trace or session. Under Expected judge result, pick one of the judge’s labels (older judges show true/false or a number field). Use the evidence and rubric to decide, rather than copying an earlier judge result.
5. Move through the examples with Prev and Next and check the labeled count. When ready, select Create labeled dataset.

Only labeled examples are included in the new dataset. The footer shows how many unlabeled examples will be excluded, so finish reviewing every example first if you need the entire source version represented.

Verify the new dataset

Open the created dataset and inspect its example count, source fields, and expected-label column for the selected judge. The column is named after the judge, for example expected_order_size for a judge named “Order size”. Check a few labels against their trace or session evidence before using them as a test baseline.

The source dataset and selected source version remain available separately.

Troubleshooting

• If Label is absent, the dataset has no trace or session field, or no version is selected.
• If the judge picker says “No compatible judges”, no judge has a production version that scores the dataset’s type (traces or sessions).
• If creation is disabled, record at least one label.
• If the new dataset has fewer examples, check whether some source examples were left unlabeled.

Next step

Align the judge against the labeled evidence before you rely on its output.