Investigate production issues

Understand how Judgment discovers and tracks unknown recurring problems across production traces.

Production issues are recurring problems discovered from trace evidence. They help you find important failure patterns that you did not already define as a judge criterion.

Issue detection reviews recent traces using your project instructions. It groups related evidence into durable issue records and favors precision over recall: it reports high-confidence patterns rather than attempting to label every trace.

Issues and behaviors answer different questions

A behavior is stable measurement infrastructure. An issue is a discovery and triage record. Do not rewrite a precise known criterion into broad issue instructions just to monitor it; create and monitor a behavior instead.

What an issue contains

An issue preserves more than a generated title. Its current record includes:

• A diagnosis and impact summary
• Priority and production activity, such as new, repeated, or resurfaced
• First-seen and last-seen times plus an evidence count
• Cited traces, spans, behaviors, or judge results
• An activity history for comments and lifecycle changes
• A status such as Needs review, Resolved, or Ignored

Open the cited evidence before accepting a diagnosis or remediation direction. The detector localizes a pattern; the traces remain the source of truth.

Detection is a recurring review, not a per-trace alert

From Issues, the initial setup asks what Judgment should look for and can start a detection run immediately. The Detection settings dialog controls:

• The instructions that focus the review
• An optional recurring schedule
• Optional automatic resolution after a period without another occurrence
• Optional Slack delivery for new issues

You can also select Run now for an on-demand review. A detection run examines a window of recent traces and may take time to complete; it is not evaluated synchronously as each trace finishes.

For immediate actions on a known behavior, exception, duration, cost, or span attribute, use Configure automations and alerts.

The issue lifecycle

Newly discovered issues enter Needs review. Triage the diagnosis and cited evidence, comment with useful context, and then:

• Resolve the issue when the current occurrence has been addressed.
• Ignore it when the pattern is not actionable for the project.
• Reopen a resolved or ignored issue when it needs active review again.

Resolved issues can resurface when new matching evidence appears. Automatic resolution can close stale issues after a configured number of days without another occurrence, while preserving their history.

Manage issues from your editor

The Judgment MCP server exposes the same issue records to coding agents in Cursor, Claude Code, Windsurf, and Codex. An agent can list and read issues, record new issues with trace evidence, update their priority or status, add comments, and resolve, ignore, or reopen them. See the production issue tools for their exact names, access levels, and inputs.

Turn discovery into a known signal

An issue often begins as an unknown pattern and becomes a criterion you want to measure continuously. Once the boundary is clear:

1. Capture representative positive and negative traces.
2. Create an Agent Judge with a precise rubric and output contract.
3. Calibrate the judge against the evidence.
4. Create and monitor a behavior for the binary or categorical output.

The issue remains the investigation history. The judge and behavior become the repeatable definition and measurement for future traffic.