Investigate a failed trace

Inspect a production trace from its failure signal through span evidence and save the case for regression testing.

Use this guide when a trace has concrete failure evidence: an exception, a known behavior result, or an output that a user reported as wrong. You will localize the failure to trace evidence and save the trace as a regression case.

Prerequisites

• Production traces in the target project
• A trace identifier, time window, customer or session identifier, exception, or behavior that can narrow the search

1. Find the trace

Open Logs > Traces. Narrow the table with the evidence you already have:

• Filter by a known behavior when a judge detected the failure.
• Filter by exception or span attributes for runtime and tool failures.
• Use customer, session, deployment, or other attributes to isolate the affected path.
• Use the trace ID directly when another alert or issue already identified it.

The table exposes input and output previews, exceptions, judge columns, automation invocations, duration, cost, session ID, customer ID, and selected attributes. Use these fields to choose a representative trace rather than only the first result.

2. Reconstruct what happened

Open the trace. Start with Conversation when the user and agent turns tell the story clearly. Switch to Tree to inspect the span tree and execution timing.

Select the span where the result diverged from the expected path. Check its input, output, attributes, status, duration, cost, and exception details. Then inspect its parents and children to determine whether the bad state originated there or arrived from an earlier model or tool call.

Do not infer a root cause from the final output alone. A plausible response can hide a failed retrieval, malformed tool result, missing context attribute, or downstream retry.

3. Inspect judge results

Open the trace’s behavior panel. For each relevant result, verify the returned value, reason, and cited evidence. Select a citation to connect the judge’s claim to the exact span that supports it.

Treat an evaluation error separately from an agent failure. An errored judge means the signal was not produced reliably; it does not prove that the agent violated the criterion.

If the result itself is wrong, record that as judge-calibration evidence. If the result is correct, continue investigating the agent path it identified.

4. Compare the surrounding population

Return to the traces table with the useful attribute, behavior, or exception filter applied. Compare a few matching and non-matching traces to answer:

• Does the failure follow one deployment, customer, tool, or input shape?
• Is the failing span absent, errored, or only returning different content?
• Is this one incident or a repeated pattern?

Keep the claim proportional to the evidence. One trace explains one execution; it does not establish prevalence by itself.

5. Save the regression case

From the trace header or overflow menu, select Add to Dataset. Choose an existing dataset or create one, map the trace into the dataset schema, and save the row.

Follow Add production traces to a dataset and confirm the new row references the production trace. Before saving the first row, map any declared expected fields or reviewer notes needed to make the failure reproducible in an offline test.

Verify the investigation

Before leaving the trace, record a concise evidence chain:

1. The observed failure signal
2. The span where the execution diverged
3. The input, output, exception, attribute, or judge reason that proves it
4. The dataset and row that preserve the case

The investigation is complete when a teammate can open the same trace, reproduce your evidence path, and find the saved regression case.

Troubleshooting

• If spans are missing or fragmented, verify trace boundaries and context propagation with the tracing data model.
• If the trace is still running, wait for its root span to complete before treating totals or judge results as final.
• If a behavior result is missing, verify the owning judge’s online mode, sampling, scope, and span triggers.
• If the failure appears across many unrelated traces and does not match a known criterion, use Production issues to investigate the recurring pattern.