Skip to Content
Judgment Labs

Set up tracing

Send your agent's traces to Judgment from Python or TypeScript.

Instrument one real agent path so its root, model calls, tool calls, input, output, and session reach the intended Judgment project. Start with supported instrumentation, then add manual spans only for work it doesn't capture.

Set up Judgment with your coding agent

Paste it into your agent from your repo's root.

Prerequisites

  • Set JUDGMENT_API_KEY and JUDGMENT_ORG_ID as described in Authentication and projects. Use an ingest API key for JUDGMENT_API_KEY: it can only send traces and observations, so it is safer than a personal API key. A personal API key also works if your role is Developer or higher. The SDK still reads JUDGMENT_ORG_ID with an ingest API key, so set both. Set JUDGMENT_API_URL only if Judgment gave you a different API URL (default https://api.judgmentlabs.ai).
  • Choose an explicit project name and deployment environment. See How a project is chosen.
  • Identify the code path that performs real agent work and what one trace should cover. See Glossary.

Load credentials in the process that runs the agent before importing tracer or client modules. A .env file alone does not load them; use your application's environment loader or inject them through its deployment configuration. Keep credentials out of source control.

Install judgeval

Install the judgeval SDK and the client used by the example in that same environment. Your framework or model client may require a newer runtime than the SDK.

Use Python 3.10 or later and judgeval 1.3.3 or later. For a new virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install judgeval openai

Use auto-instrumentation

Two integration families cover most applications:

IntegrationCapturesUse it when
Agent frameworkAgent steps, tool calls, and the model calls exposed by the frameworkThe application uses a supported framework. Start here.
Model providerProvider calls, token usage, and costThe application calls a supported provider client directly.

Enable the integration that matches the application. Open one app root around the work so integration spans become its children, then add Tracer.observe only for missing work such as custom tools and retrievals.

Record each model call once. Don't wrap a client that the framework integration already records, or enable provider instrumentation on top of it. Either combination duplicates the model span, tokens, and cost.

TypeScript Tracer.wrap supports OpenAI clients only. Python Tracer.wrap supports OpenAI, Anthropic, Together, and Google GenAI (google.genai.Client). Tracer.init doesn't install a global OpenTelemetry provider, so a third-party instrumentor exports nothing until you register it with Tracer.registerOTELInstrumentation(...). See OTel instrumentation libraries.

TypeScript OpenTelemetry auto-instrumentation requires all three of these:

  • A file the runtime loads before the application.
  • Instrumentations registered with Judgment's tracer provider before the application loads the modules they patch.
  • A CommonJS build, because the instrumentations patch modules when they are required.

Tracer.registerOTELInstrumentation() needs Judgment's tracer provider to exist. Tracer.init() creates it, but only after an await, and a CommonJS file can't use top-level await. So create the provider with JudgmentTracerProvider.getInstance(), register the instrumentations, and then start Tracer.init():

instrumentation.ts
import { JudgmentTracerProvider, Tracer } from "judgeval";
import { OpenAIInstrumentation } from "@opentelemetry/instrumentation-openai";

JudgmentTracerProvider.getInstance();
Tracer.registerOTELInstrumentation(new OpenAIInstrumentation());

export const tracerReady = Tracer.init({
  projectName: "lunch-order-agent",
  environment: "production",
});

Import this file first in the entrypoint, or preload it with the runtime's require flag (node --require ./instrumentation.js). Spans that start before Tracer.init() resolves are not recorded, so await tracerReady before the application handles its first request. If Tracer.registerOTELInstrumentation() runs before the provider exists, the SDK logs No active tracer provider. Instrumentation was not registered. and the instrumentation exports nothing. For bundlers, also follow Serverless and Cloudflare Workers.

Initialize tracing first

Initialize once in every process that creates spans, before constructing wrapped clients, observed functions, workers, or the application.

tracer.py
from judgeval import Tracer

Tracer.init(
    project_name="lunch-order-agent",
    environment="production",
)

Use environment to distinguish development, staging, and production traces within a project. Traces without an environment show as No environment.

Tracer.init looks up the project by name and creates it if no project has that name (developer role required), so a typo in the project name sends traces to a new, empty project. With a viewer-role key the lookup fails, and the SDK logs a warning and exports nothing. Neither case raises an exception. Log the selected project and deployment environment at startup, but never log the API key.

To turn export off, skip Tracer.init, or leave JUDGMENT_API_KEY, JUDGMENT_ORG_ID, or the project name unset. The SDK logs a warning, and observed functions still run, untraced.

Trace the agent path

This example uses the lunch-order agent and assumes no framework integration. It creates one root, traces one tool, and wraps one supported model client. Keep only the pieces your selected integration doesn't already provide. Set your and choose a model available to your account before running it.

Every span is named explicitly. Production bundlers can strip function names, which otherwise leaves blank names in the trace view.

The example's chat-123 identifies one conversation. Reuse that ID for follow-ups that continue its context, and choose a new ID for a new chat. Leave session_id unset for work outside a conversation. See Session guidance.

import tracer  # initializes tracing before the client is created

from judgeval import Tracer
from openai import OpenAI

client = Tracer.wrap(OpenAI())

@Tracer.observe(span_type="tool", span_name="search_nearby_restaurants")
def search_nearby_restaurants(cuisine: str) -> list[str]:
    return ["Olive & Thyme", "Falafel House", "Cedar Grill"]

@Tracer.observe(span_type="agent", span_name="lunch_order")
def handle_request(session_id: str, message: str) -> str:
    Tracer.set_session_id(session_id)
    restaurants = search_nearby_restaurants("mediterranean")
    response = client.chat.completions.create(
        model="gpt-5.2",
        messages=[
            {"role": "system", "content": f"Restaurants nearby: {restaurants}"},
            {"role": "user", "content": message},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(handle_request("chat-123", "order me mediterranean food to feed 30 hungry engineers"))
    Tracer.shutdown()

Save the initialization above as tracer.py and this agent as agent.py, then run it in the activated environment:

python agent.py

Activate the root span

A span becomes the parent of new work only while it is active:

LanguageActive boundary
PythonTracer.observe, Tracer.span(), or Tracer.start_as_current_span()
TypeScriptTracer.observe, Tracer.startActiveSpan, or Tracer.getOTELTracer().startActiveSpan

In TypeScript, Tracer.startSpan returns a span without activating it. Tracer.setSessionId() then has no active span to update, and later work doesn't become a child. Raw OpenTelemetry context.with and trace.setSpan also don't activate the SDK's own context. Use an active boundary from the table.

Python has the same lifecycle distinction: Tracer.start_span() returns a span that you must manage. Prefer an active boundary whenever work should inherit that span.

Long-lived background work started while a request span is active can inherit the request forever. Start the background loop outside any active span and give each unit its own root.

Flush completed roots

Judgment starts evaluation when the completed root arrives, so every child must be sent before the root. End the root with a real duration, output, and status before flushing. Spans wait in a batch queue for about five seconds; anything still queued when a process stops is lost.

Runtime shapeWhere to flush
Script or CLIAfter the final root, then shut down before exit. shutdown is synchronous in Python and a promise in TypeScript.
Serverless handlerAfter the root ends and before the runtime can freeze.
WorkerAfter each job if the process can stop before the next export.
Long-running serverAt graceful shutdown; flush after each root if the process can be killed or restarted without warning.
Durable loopBefore saving the checkpoint for each completed part.
result = run_job(job)  # the observed root ends when this returns
Tracer.force_flush()

Verify traces

Check the startup logs

Run real agent work with tracing initialized, and read the startup logs. project_name not provided, api_key not provided, organization_id not provided, or Project '...' not found means nothing was exported.

Find the trace

Confirm the trace reached the expected project and environment. Storage is asynchronous, so give it a moment.

Check what the trace recorded

Open the trace and confirm the named app root contains the expected model and tool spans, input, output, and session ID.

Read trace health

Read the trace health results. Judgment checks root structure, nesting, timing, input/output, LLM metadata, and session ID consistency. A check with no status is still pending, not passing. See Trace health.

Check for extra traces

Confirm plumbing didn't create extra traces and each request produced one expected trace.

Test a hard case

Test one hard case: restart, stream abort, worker retry, approval resume, handled tool failure, or serverless cold call.

A coding agent can read the same health results with get_trace_health, get_trace_health_summary, and list_trace_health_failures through the Judgment MCP server.

Troubleshooting

SymptomCheck
No project or tracesCheck the startup warnings, credentials, organization, key role, network access, and whether real traced code ran. A viewer-role key, a network error, or a missing credential disables export without an exception.
Project exists but is emptyCheck that the process handling real traffic received the tracing settings and initialization code.
Traces reached the wrong projectThe name in code doesn't match the project you're looking at, so the SDK created a new one. Fix the name, and log the active project before the root starts.
Export rejected with 413A span has more than 8 MiB of attributes, and the whole batch was rejected. See Ingest contract.
Export rejected with 402The organization hit its usage limit.
Root has zero duration or no outputEnd the root before flushing or stopping the worker.
Named root has no integration children or sessionCheck initialization order and whether the root was active.
One request breaks into roots across servicesPropagate OpenTelemetry context between services. See Distributed services.
Model spans or cost appear twiceChoose manual or automatic LLM instrumentation, not both.
Returned failures stay greenMark handled failures explicitly. See Mark handled failures.
Test traces appear with production trafficSet the SDK's environment option for development and test runs within the same project. See Environment.

Next step

Add context once the trace structure is healthy. Tracing patterns covers streaming, queues, serverless, and distributed services. The Tracer SDK Reference contains the complete Python and TypeScript APIs.

On this page