Labels as indexes
Why judge labels let the Judgment Agent answer questions without rereading every trace.
Say you want to know how often users get frustrated with your agent, and what causes it.
If you point a coding agent at 100 traces and ask, it’ll read a subset of them in full, including all the raw search results and metadata that have nothing to do with how the user felt, and then give you an answer. That works at small scale, but with long traces or thousands of them, it gets slow, expensive and less accurate.
A judge reads the trace once, when it arrives
A judge is an agent that evaluates a trace or a whole session against a rubric you write, then applies a label. “How does the user feel about the agent?” might apply Positive, Negative or Neutral. “How big are orders?” might apply Small, Medium or Large.
The judge does the expensive reading up front. An Agent judge queries the trace’s spans itself: it pulls span metadata first, picks the spans that matter for the rubric, reads only those inputs and outputs, and cites the spans it used in its reasoning. You don’t have to tell it where in the trace to look.
Run a few judges that each answer a different question, and every trace in your project ends up tagged along several dimensions.
Labels become indexes
Because labels are computed before you ask anything, they work like indexes on your trace data.
When you ask the Judgment Agent “How often do users get frustrated, and what causes it?”, it doesn’t have to read every trace to figure out whether the user was frustrated. It finds the sessions labeled Negative, then looks at the labels other judges applied to those same traces (order size, cuisine, whether the request was completed) to see what went wrong. It can still open individual traces when it needs evidence for a specific claim.
The analysis is cheaper and faster than handing raw traces to a coding agent, and the answers stay detailed as volume grows. Your coding agent can query the same labels with SQL.
With judges for sentiment, order size, cuisine and request completion on the lunch-order agent, “Which kinds of orders leave users unhappy, and were their requests completed?” becomes a lookup across labels instead of a read through a week of traces. Cookbook: Data analysis sets up those judges.
What makes a good index judge
- Ask one question per judge. “How does the user feel?” and “Was the request completed?” are two judges, not one. Narrow questions give you labels you can combine.
- Pick the scope that matches the question. Sentiment and completion play out over a conversation, so judge the session. Order size is a fact about one request, so judge the trace.
- Name labels the way you’d search for them. You’ll be asking questions in plain English, so Negative beats sentiment_score_low.
- Give each judge two or more named choices, like Completed and Not completed. Each choice becomes a label you can filter on.
Labels apply to traces that arrive after a judge is turned on, so set up the judges before you need the answers. Run a judge on live traffic covers turning them on.