Add production traces to a dataset

Map production trace evidence into schema-enforced dataset rows for review and regression testing.

Use this guide after a production trace reveals a failure, edge case, or useful success. You will save the trace as a schema-enforced row that remains available for calibration and regression testing.

Prerequisites

A completed production trace in the target project
An existing non-legacy dataset, or permission to create one
A dataset schema that includes one trace field, unless the target dataset is still an empty draft

If you have not localized the failure yet, follow Investigate a failed trace first.

Add the trace

Select the evidence

Open Logs > Traces. Select one or more traces in the table and choose Add to Dataset from the selection bar.

For one open trace, use Add to Dataset in the trace header or overflow menu.

Choose the dataset

Select an existing dataset. To start a new collection, choose Create New Dataset, enter a name, and continue with the new draft selected.

Legacy datasets do not appear because they are read-only.

Map trace fields into the row

Map source trace fields to the dataset fields. A defined schema fixes the target names and types. For an empty draft, the mapping defines the schema when you save the first trace.

The LLM Eval mapping is a useful starting point for input and actual_output. Add fields for expected output, labels, or context only when the dataset schema declares them.

The trace pointer is filled automatically and appears in the preview. Do not map the trace ID into an ordinary string field.

Review and save

Inspect Example Preview. Confirm that the values, field names, and types match the target schema, then select Done.

Judgment creates one example per selected trace, preserves the source trace link, and creates a new dataset version for the batch.

Verify the saved evidence

Open the dataset from the success message or from Datasets. Confirm that:

The latest version contains the new row.
The mapped input, output, and context fields match the source trace.
The trace field contains the source trace ID and opens the saved trace evidence.

The production case is now durable test input rather than only a row in a moving time window.

Troubleshooting

If the dataset is missing from the picker, it may be a read-only legacy dataset.
If Done is disabled, select a dataset and resolve empty, duplicate, or type-incompatible target fields.
If the preview is still loading, wait before saving an empty draft. Judgment uses the preview values to infer its field types.
If the schema needs another field after its first example, create a new dataset with the intended schema and add the trace there.
If the saved trace does not contain the evidence you expected, return to the production trace and verify its span boundaries before treating it as a regression case.

Next step

Turn the production failure into a regression test.