---
title: "Investigate a failed trace"
description: "Inspect a production trace from its failure signal through span evidence and save the case for regression testing."
sidebar:
  label: "Investigate a failed trace"
seo:
  title: "Investigate a Failed Trace | Judgment How-to"
  description: "Find a failed production trace, inspect its spans and judge results, and add the evidence to a dataset for regression testing."
---

Use this guide when a trace has concrete failure evidence: an exception, a
known behavior result, or an output that a user reported as wrong. You will
localize the failure to trace evidence and save the trace as a regression case.

## Prerequisites

- Production traces in the target project
- A trace identifier, time window, customer or session identifier, exception,
  or behavior that can narrow the search

## 1. Find the trace

Open **Logs > Traces**. Narrow the table with the evidence you already
have:

- Filter by a known behavior when a judge detected the failure.
- Filter by exception or span attributes for runtime and tool failures.
- Use customer, session, deployment, or other attributes to isolate the
  affected path.
- Use the trace ID directly when another alert or issue already identified it.

The table exposes input and output previews, exceptions, judge columns,
automation invocations, duration, cost, session ID, customer ID, and selected
attributes. Use these fields to choose a representative trace rather than only
the first result.

## 2. Reconstruct what happened

Open the trace. Start with **Conversation** when the user and agent turns tell
the story clearly. Switch to **Tree** to inspect the span tree and execution
timing.

Select the span where the result diverged from the expected path. Check its
input, output, attributes, status, duration, cost, and exception details. Then
inspect its parents and children to determine whether the bad state originated
there or arrived from an earlier model or tool call.

Do not infer a root cause from the final output alone. A plausible response can
hide a failed retrieval, malformed tool result, missing context attribute, or
downstream retry.

## 3. Inspect judge results

Open the trace's behavior panel. For each relevant result, verify the returned
value, reason, and cited evidence. Select a citation to connect the judge's
claim to the exact span that supports it.

Treat an evaluation error separately from an agent failure. An errored judge
means the signal was not produced reliably; it does not prove that the agent
violated the criterion.

If the result itself is wrong, record that as judge-calibration evidence. If
the result is correct, continue investigating the agent path it identified.

## 4. Compare the surrounding population

Return to the traces table with the useful attribute, behavior, or exception
filter applied. Compare a few matching and non-matching traces to answer:

- Does the failure follow one deployment, customer, tool, or input shape?
- Is the failing span absent, errored, or only returning different content?
- Is this one incident or a repeated pattern?

Keep the claim proportional to the evidence. One trace explains one execution;
it does not establish prevalence by itself.

## 5. Save the regression case

From the trace header or overflow menu, select **Add to Dataset**. Choose an
existing dataset or create one, map the trace into the dataset schema, and save
the row.

Follow [Add production traces to a
dataset](/documentation/datasets/production-traces) and confirm the new row
references the production trace. Before saving the first row, map any declared
expected fields or reviewer notes needed to make the failure reproducible in
an offline test.

## Verify the investigation

Before leaving the trace, record a concise evidence chain:

1. The observed failure signal
2. The span where the execution diverged
3. The input, output, exception, attribute, or judge reason that proves it
4. The dataset and row that preserve the case

The investigation is complete when a teammate can open the same trace,
reproduce your evidence path, and find the saved regression case.

## Troubleshooting

- If spans are missing or fragmented, verify trace boundaries and context
  propagation with the [tracing data model](/documentation/tracing).
- If the trace is still running, wait for its root span to complete before
  treating totals or judge results as final.
- If a behavior result is missing, verify the owning judge's online mode,
  sampling, scope, and span triggers.
- If the failure appears across many unrelated traces and does not match a
  known criterion, use [Production
  issues](/documentation/monitoring/production-issues) to investigate the
  recurring pattern.