Cookbook: Root-cause an Issue

Take one Issue from the inbox to a shipped fix and a judge that watches for it.

Your agent has been in production for a week. Nobody has complained and the dashboards are green, but you suspect it’s failing somewhere you can’t see.

Which Issue should I look at first?

Open Issues. The inbox shows Needs review Issues, sorted by Severity. Near the top is a High severity Issue, trend First seen, titled “Menu lookups time out and the agent silently drops the restaurant”.

Read its three short sections before anything else. On this Issue they’d say something like:

• Description: get_menu returns 504 Gateway Timeout for one restaurant, the agent retries once, then drops the restaurant from the shortlist without telling the user.
• Impact: requests for larger catering orders end up with fewer options; the user never learns a restaurant was skipped.
• Consider: tell the user when a restaurant couldn’t be checked, or retry with a longer timeout before dropping it.

What actually happened in the trace?

Under Evidence, each row is one trace, with what happened, where, and when. Open one. It’s the request from the example trace, “order me mediterranean food to feed 30 hungry engineers”, and the failure is in the middle:

• tool get_menu x3: Olive & Thyme and Falafel House return menus, Cedar Grill fails (504 Gateway Timeout)
• tool get_menu retry: Cedar Grill fails again
• llm call: drop Cedar Grill and continue with the other two

The final response recommends Olive & Thyme, $612 total, delivered by 12:15 PM, and never mentions Cedar Grill. The response looks fine, which is why nobody noticed: the failure only shows up mid-trajectory.

Open a second evidence trace and make sure it fails the same way. If it fails for a different reason, say so in a comment under Activity; the diagnosis may be grouping two problems.

How big is it?

Before you prioritize a fix, get numbers. Select the Issue in the inbox and choose Ask Judgment:

For the Issue "Menu lookups time out and the agent silently drops the
restaurant": over the last 7 days, how many traces had a get_menu call that
returned a 504? Which restaurants were affected? In how many of those traces
did the final response mention that a restaurant was skipped? Cite traces.

The agent searches your traces with Judgment’s virtual SQL (vsql), reads the spans it needs, and answers with citations you can click. You might learn the timeouts all come from two restaurants, or that they spike at 11:30 AM when everyone orders. Either one would change the fix.

The same question works from Codex, Claude Code or Cursor with the Judgment MCP server connected. The agent uses get_issue to read the Issue and sql_query to count. See Connect your coding agent.

Who fixes it, and how?

Choose Fix with on the Issue and pick your coding agent. The prompt includes the diagnosis and cited traces; see Triage and fix for exactly what it contains.

For this Issue, a reasonable fix has two parts: retry get_menu with backoff before giving up, and when a restaurant still can’t be checked, say so in the response (“I couldn’t load Cedar Grill’s menu, so I left it out”). Your coding agent proposes the change, and you review and ship it like any other pull request. Judgment doesn’t merge or deploy anything.

How do I make sure it doesn’t come back?

Two things, and both are manual today.

Keep the failing cases

Open an evidence trace and use Add to Dataset to save it to a dataset of regression examples, or ask the agent to create a dataset and mention the Issue with @ (“Turn recent failures into test cases”). Run your agent over that dataset before and after the fix and compare the runs. Run a test from your code shows how.

Watch production with a judge

Back on the Issue, choose Review, then Monitor with Judge. This opens a Judgment Agent chat that creates a judge for this failure, or reuses one you have. You can steer it:

Create a trace-level judge: "Did a tool fail to return data the agent needed,
and did the agent tell the user?" Labels: No tool failure, Failed and told
user, Failed silently. Turn on online monitoring for production.

The Issue becomes Monitored once the agent links that judge and the Failed silently label to it, which needs the judge running on production traffic (Triage and fix has the exact condition). You can then see the rate of silent failures per day, filter traces by that label, and send it to Slack with a judge action.

How do I know the fix worked?

After the fix ships, the judge’s label rate for Failed silently should drop toward zero on new traffic. See Where labels live. A Monitored Issue doesn’t go back to Regressed (what happens to an Issue), so the judge is the signal.

Do it from your coding agent

With the Judgment MCP server connected, paste this into Codex, Claude Code or Cursor:

Using Judgment, list the Needs review issues in the lunch-order-agent project
and pick the highest-severity one. Read the issue and two of its cited traces,
tell me whether the diagnosis holds, and if the cause is in this repo, propose
the smallest fix. Don't change the issue's status until I confirm.