Glossary

The terms you'll see across Judgment, defined.

Each entry ends with where the term shows up in the app and what it’s called in code.

Your agent’s data

Trace

One run of your agent: every model call, tool call, input and output, with timing, cost and errors. In the app: the Traces tab. In code: everything under one root span, identified by trace_id.

Span

One step inside a trace, such as a model call, a tool call, or a function you decorate. Spans nest, so a trace reads as a tree. In the app: the trace tree and the Spans tab. In code: Tracer.observe(span_type="tool"); the kind is stored as judgment.span_kind (llm, tool, agent, function, or span by default).

Session

The traces from one conversation, such as a request and the user’s follow-up reply. In the app: the Sessions tab. In code: Tracer.set_session_id(...) while the root span is active. Judgment reads judgment.session_id from the root span only.

Project

One agent. Its traces, judges, Issues and datasets all live in the project. In code: project_name in Tracer.init. If no project has that name, the SDK creates one (this needs a developer role or higher); see How a project is chosen.

Environment

Development, staging and production traffic inside one project. Judges can be scoped to environments. In code: Tracer.init(..., environment="production"), stored as the deployment.environment.name resource attribute. Traces without one show as No environment.

Judging and finding problems

Judge

Reads a trace or a session and applies one of two or more labels you define. An Agent judge follows a rubric you write, a Code judge runs Python code you upload as a bundle, and a Condition judge checks conditions on the trace. A judge can also take actions when it applies a label: post to Slack, send an email, add the trace to a dataset, or run another judge. In the app: Judges, with run settings, evaluation method and actions on the judge’s Details tab. The evaluation methods are called Agentic Judge, Custom code and Deterministic conditions. In code and MCP: create Agent and Condition judges in the app, with the Judgment Agent, or with the MCP tool create_judge. Code judges can’t be created in the app; upload a bundle with the API or create_judge (see Code judge bundle).

Label

The value a judge applies, such as Negative from a judge that rates user sentiment. Labels act as indexes for the Judgment Agent. In code: API and SQL names still say behavior; it means label (behavior_id, config.behaviors).

Issue

A recurring problem the Issues agent found in your traces, with a severity, a diagnosis and the traces behind it. Issues need traces only, no judges. In the app: Issues, with statuses Needs review, Monitored, Inactive and Dismissed. In MCP: list_issues, get_issue, update_issue.

Testing changes

Test suite

A versioned set of cases your agent runs on. Each case has an input for your agent and, in expected, the judges to run on its trace and the result each should return. You edit cases in a draft and publish it as the next version. In code: client.test_suites in Python, judgeval.testSuites in TypeScript. See Test your agent.

Test run

One pass of your agent over a test suite’s published version. Your code runs the agent once per case and tags each case’s root span with judgment.test.run_id and judgment.test.case_id; Judgment scores each case when its trace arrives. A case passes when every check in its expected passes. In code: client.test_runs.start(...) and Tracer.test_case(...) in Python, judgeval.testRuns.start(...) and Tracer.testCase(...) in TypeScript.

Dataset

A versioned set of examples, such as saved traces or the inputs you want to test your agent on. You can label a dataset against a judge, and test config runs pin the dataset version they used. In the app: Datasets & Tests. In code: client.datasets. In MCP: add_examples_to_dataset.

Test config (legacy)

A dataset plus the judges that grade each run. Deprecated in favor of test suites; through the SDK and API, only organizations that used test configs before test suites launched can run them. In the app: Datasets & Tests. In code: client.offline_tests.run(...) in Python, client.offlineTests.run(...) in TypeScript. See Run a test config from your code.

Billing

Judgment Units (JU)

The unit Judgment bills in. See Billing and plans for what each plan includes.