Cookbook: Data analysis

Set up judges so the Judgment Agent can answer questions like why users got frustrated last week.

Using the Judgment Agent alongside tracing and judges gives you answers to complex questions about your agent, like this one about the lunch-order agent:

Identify when users were frustrated in the past week, tracking trends across size of orders and types of restaurants, and let me know what caused their frustration, whether these problems are new or recurring, and if their problem ended up being solved.

This cookbook sets up Judgment to answer that question, and others like it.

If you haven’t sent traces yet, start with Set up tracing.

Set up the judges

Each judge below labels every new trace, so the Judgment Agent can look up labels instead of rereading traces. Labels as indexes explains why. Each judge is a question plus its labels:

The fastest way to create all four is to ask the Judgment Agent from your project’s home page:

Create four judges for this project:
1. "User sentiment": session scoped, with Positive, Negative, Neutral.
   Judge how the user feels about the agent across the whole conversation.
2. "Order size": trace scoped, with Small (under $200), Medium ($200 to $500),
   Large (over $500), based on the estimate_order total.
3. "Cuisine": trace scoped, with one choice per cuisine our users order.
4. "Request completed": session scoped, with Completed and Not completed. Not completed
   when the user's original request was never fulfilled, including when the agent
   dropped an option without saying so.
Turn on online monitoring for production traffic.

The agent asks before it creates anything. You can also create each judge yourself with New Judge in Judges; see Create a judge.

Then check Run settings on each judge’s Details tab:

• Online monitoring is on and the version is published to production.
• Session scoring is on for the two session judges.
• Environments includes the environment your production traffic uses.

Run a judge on live traffic covers every run setting.

Let traffic come in

Judges label traces as they arrive. They don’t go back over traces you already have, so give it a day or a week of real traffic before you ask a question that spans a week.

While you wait, open a few labeled traces and check the calls. If User sentiment marks an obviously annoyed user as Neutral, fix the rubric now, before you build analysis on top of it. Align a judge is the structured way to do this.

Ask the question

Open the Judgment Agent and paste the question from the top of this page. Deep Research mode is a good fit for questions like this one, which take several queries to answer.

Here’s roughly what the agent does:

1. Finds sessions labeled Negative by User sentiment in the past week.
2. Breaks them down by the Order size and Cuisine labels on their traces, so you can see whether frustration clusters on Large orders or on one cuisine.
3. Checks Request completed on the same sessions to see whether the problem ended up being solved.
4. Opens a few of those traces to find the cause, like Cedar Grill’s get_menu timing out and the agent dropping it without telling the user.
5. Compares with earlier weeks to tell you whether it’s new or recurring.

Under the hood, the agent reads labels with SQL. Judgment’s virtual SQL (vsql) keeps every judge result in telemetry.scores. You can run the same queries from Codex, Claude Code or Cursor with the MCP sql_query tool. See Query with SQL.

Track trends over time

A question like this is usually worth asking every week. There are two ways to keep it running:

• Save the question as a Scheduled Task’s instructions, set it to repeat weekly, and turn on Post to Slack. See Scheduled Tasks.
• If you want to hear about every frustrated session as it happens, add an action on User sentiment that alerts when it applies Negative. See Judge actions.

If a cause shows up week after week, check Issues. Detection looks for recurring problems on its own, without any judges, and may already be tracking it.

What’s manual

• You decide which questions deserve a judge. Judgment can suggest judges (Suggest judges to create in New Judge), but you pick.
• Judges only label new traffic, so a new question needs time to collect data.
• Fixing the cause is up to you and your coding agent. Judgment shows you where the problem is and which traces prove it.