Run a judge on live traffic

Turn on online monitoring under Run settings, choose which traffic the judge evaluates, and run a judge on traces you already have.

A judge only labels live traffic after you turn on online monitoring. Everything on this page is under Run settings on the judge’s Details tab, except running a judge on traces you already have.

Turn on online monitoring

1. Open Judges, pick the judge, and open Run settings on the Details tab.
2. Turn on Online monitoring (“Automatically evaluate incoming traces”).
3. Set the options below, then select Save and Publish to production.

Online monitoring runs the judge’s production version. A draft doesn’t label anything until you publish or promote it.

Run settings appears once the judge exists. For a Code judge you can also set these options through MCP; see Code judge bundle.

If the judge isn’t labeling, Run settings says so: “This judge is not tagging incoming traces.” That appears when monitoring is off or the sampling rate is 0%.

Choose which traffic gets judged

Environments

Select the environments to evaluate, or all of them. You can’t save with online monitoring on until you select them. A judge scoped to production ignores your local runs.

Traces without an environment show as No environment. To set environments in your code, see Add context.

Run conditions

Leave them empty to evaluate every trace. Add conditions to evaluate only the traces that match:

• Duration (seconds) and LLM cost (USD) compare the trace’s total with a number (>, ≥, <, ≤, =).
• Error checks the trace’s error: exists, equals, does not equal, or contains an error message.
• A Span group checks one span. Under Span scope, select Root span or Any single span, then add Span Name or Span Attribute conditions (with an Attribute Key).

Every condition in a span group must match the same span. Combine conditions and span groups with AND, OR, and parentheses.

For the lunch-order agent, a judge about menu failures could use a span group with Any single span and Span Name equals get_menu, so it only runs on traces that looked up a menu.

Sampling rate

The sampling rate is the percentage of incoming traces (or sessions, with session scoring on) to evaluate, from 0 to 100%. Lower it to control cost on high-volume projects. Judge runs use Judgment Units. See Billing and plans.

Session scoring

Turn on Session scoring to evaluate the whole session instead of each trace. Then select when a session gets evaluated:

Use session scoring for questions that play out over a conversation, like “How does the user feel about the agent?” Condition judges can’t use session scoring.

Run a judge on traces you already have

Online monitoring only labels traces that arrive after you turn it on. To label traces or sessions you already have:

1. Open the traces (or sessions) table and select the ones to evaluate.
2. Select Run Judge.
3. Pick the judge and select Run Judge.

The list only shows judges that match what you selected: trace judges for traces, and judges with session scoring on for sessions. The judge’s production version runs on each item. Its environments, run conditions and sampling rate still apply, so a trace those settings exclude isn’t evaluated. These runs don’t send the judge’s actions.

To evaluate a specific span from your code instead, call Tracer.async_evaluate with the judge’s name inside an observed function. The evaluation is queued when the span ends and runs in Judgment:

from judgeval import Tracer

Tracer.init(project_name="lunch-order-agent")


@Tracer.observe(span_type="agent", span_name="recommend_order")
def recommend_order(request: str) -> str:
    recommendation = "Cedar Grill for 30 people, total $480."
    Tracer.async_evaluate(
        "Order size",
        {"input": request, "actual_output": recommendation},
    )
    return recommendation


print(recommend_order("Lunch for 30 people on Friday"))
Tracer.shutdown()

If no labels appear

• Check that Online monitoring is on and the version you edited is the production version.
• Check the sampling rate is above 0%.
• Check the environment scope matches the environment your traces are sent with.
• Check the run conditions. A span group set to Root span never matches a nested span; switch it to Any single span.
• Labels apply to traces that arrive after monitoring is on. For older traces, run the judge on them.
• With session scoring, a session isn’t evaluated until the inactivity window or trace count is reached.