Set up tracing
Send your agent's traces to Judgment from Python or TypeScript.
Instrument one real agent path so its root, model calls, tool calls, input, output, and session reach the intended Judgment project. Start with supported instrumentation, then add manual spans only for work it doesn't capture.
Set up Judgment with your coding agent
Paste it into your agent from your repo's root.
Prerequisites
- Set
JUDGMENT_API_KEYandJUDGMENT_ORG_IDas described in Authentication and projects. Use an ingest API key forJUDGMENT_API_KEY: it can only send traces and observations, so it is safer than a personal API key. A personal API key also works if your role is Developer or higher. The SDK still readsJUDGMENT_ORG_IDwith an ingest API key, so set both. SetJUDGMENT_API_URLonly if Judgment gave you a different API URL (defaulthttps://api.judgmentlabs.ai). - Choose an explicit project name and deployment environment. See How a project is chosen.
- Identify the code path that performs real agent work and what one trace should cover. See Glossary.
Load credentials in the process that runs the agent before importing tracer or client modules. A .env file alone does not load them; use your application's environment loader or inject them through its deployment configuration. Keep credentials out of source control.
Install judgeval
Install the judgeval SDK and the client used by the example in that same environment. Your framework or model client may require a newer runtime than the SDK.
Use Python 3.10 or later and judgeval 1.3.3 or later. For a new virtual environment:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install judgeval openaiUse auto-instrumentation
Two integration families cover most applications:
| Integration | Captures | Use it when |
|---|---|---|
| Agent framework | Agent steps, tool calls, and the model calls exposed by the framework | The application uses a supported framework. Start here. |
| Model provider | Provider calls, token usage, and cost | The application calls a supported provider client directly. |
Enable the integration that matches the application. Open one app root around the work so integration spans become its children, then add Tracer.observe only for missing work such as custom tools and retrievals.
Record each model call once. Don't wrap a client that the framework integration already records, or enable provider instrumentation on top of it. Either combination duplicates the model span, tokens, and cost.
TypeScript Tracer.wrap supports OpenAI clients only. Python Tracer.wrap supports OpenAI, Anthropic, Together, and Google GenAI (google.genai.Client). Tracer.init doesn't install a global OpenTelemetry provider, so a third-party instrumentor exports nothing until you register it with Tracer.registerOTELInstrumentation(...). See OTel instrumentation libraries.
TypeScript OpenTelemetry auto-instrumentation requires all three of these:
- A file the runtime loads before the application.
- Instrumentations registered with Judgment's tracer provider before the application loads the modules they patch.
- A CommonJS build, because the instrumentations patch modules when they are required.
Tracer.registerOTELInstrumentation() needs Judgment's tracer provider to exist. Tracer.init() creates it, but only after an await, and a CommonJS file can't use top-level await. So create the provider with JudgmentTracerProvider.getInstance(), register the instrumentations, and then start Tracer.init():
import { JudgmentTracerProvider, Tracer } from "judgeval";
import { OpenAIInstrumentation } from "@opentelemetry/instrumentation-openai";
JudgmentTracerProvider.getInstance();
Tracer.registerOTELInstrumentation(new OpenAIInstrumentation());
export const tracerReady = Tracer.init({
projectName: "lunch-order-agent",
environment: "production",
});Import this file first in the entrypoint, or preload it with the runtime's require flag (node --require ./instrumentation.js). Spans that start before Tracer.init() resolves are not recorded, so await tracerReady before the application handles its first request. If Tracer.registerOTELInstrumentation() runs before the provider exists, the SDK logs No active tracer provider. Instrumentation was not registered. and the instrumentation exports nothing. For bundlers, also follow Serverless and Cloudflare Workers.
Initialize tracing first
Initialize once in every process that creates spans, before constructing wrapped clients, observed functions, workers, or the application.
from judgeval import Tracer
Tracer.init(
project_name="lunch-order-agent",
environment="production",
)Use environment to distinguish development, staging, and production traces within a project. Traces without an environment show as No environment.
Tracer.init looks up the project by name and creates it if no project has that name (developer role required), so a typo in the project name sends traces to a new, empty project. With a viewer-role key the lookup fails, and the SDK logs a warning and exports nothing. Neither case raises an exception. Log the selected project and deployment environment at startup, but never log the API key.
To turn export off, skip Tracer.init, or leave JUDGMENT_API_KEY, JUDGMENT_ORG_ID, or the project name unset. The SDK logs a warning, and observed functions still run, untraced.
Trace the agent path
This example uses the lunch-order agent and assumes no framework integration. It creates one root, traces one tool, and wraps one supported model client. Keep only the pieces your selected integration doesn't already provide. Set your and choose a model available to your account before running it.
Every span is named explicitly. Production bundlers can strip function names, which otherwise leaves blank names in the trace view.
The example's chat-123 identifies one conversation. Reuse that ID for follow-ups that continue its context, and choose a new ID for a new chat. Leave session_id unset for work outside a conversation. See Session guidance.
import tracer # initializes tracing before the client is created
from judgeval import Tracer
from openai import OpenAI
client = Tracer.wrap(OpenAI())
@Tracer.observe(span_type="tool", span_name="search_nearby_restaurants")
def search_nearby_restaurants(cuisine: str) -> list[str]:
return ["Olive & Thyme", "Falafel House", "Cedar Grill"]
@Tracer.observe(span_type="agent", span_name="lunch_order")
def handle_request(session_id: str, message: str) -> str:
Tracer.set_session_id(session_id)
restaurants = search_nearby_restaurants("mediterranean")
response = client.chat.completions.create(
model="gpt-5.2",
messages=[
{"role": "system", "content": f"Restaurants nearby: {restaurants}"},
{"role": "user", "content": message},
],
)
return response.choices[0].message.content
if __name__ == "__main__":
print(handle_request("chat-123", "order me mediterranean food to feed 30 hungry engineers"))
Tracer.shutdown()Save the initialization above as tracer.py and this agent as agent.py, then run it in the activated environment:
python agent.pyActivate the root span
A span becomes the parent of new work only while it is active:
| Language | Active boundary |
|---|---|
| Python | Tracer.observe, Tracer.span(), or Tracer.start_as_current_span() |
| TypeScript | Tracer.observe, Tracer.startActiveSpan, or Tracer.getOTELTracer().startActiveSpan |
In TypeScript, Tracer.startSpan returns a span without activating it. Tracer.setSessionId() then has no active span to update, and later work doesn't become a child. Raw OpenTelemetry context.with and trace.setSpan also don't activate the SDK's own context. Use an active boundary from the table.
Python has the same lifecycle distinction: Tracer.start_span() returns a span that you must manage. Prefer an active boundary whenever work should inherit that span.
Long-lived background work started while a request span is active can inherit the request forever. Start the background loop outside any active span and give each unit its own root.
Flush completed roots
Judgment starts evaluation when the completed root arrives, so every child must be sent before the root. End the root with a real duration, output, and status before flushing. Spans wait in a batch queue for about five seconds; anything still queued when a process stops is lost.
| Runtime shape | Where to flush |
|---|---|
| Script or CLI | After the final root, then shut down before exit. shutdown is synchronous in Python and a promise in TypeScript. |
| Serverless handler | After the root ends and before the runtime can freeze. |
| Worker | After each job if the process can stop before the next export. |
| Long-running server | At graceful shutdown; flush after each root if the process can be killed or restarted without warning. |
| Durable loop | Before saving the checkpoint for each completed part. |
result = run_job(job) # the observed root ends when this returns
Tracer.force_flush()Verify traces
Check the startup logs
Run real agent work with tracing initialized, and read the startup logs. project_name not provided, api_key not provided, organization_id not provided, or Project '...' not found means nothing was exported.
Find the trace
Confirm the trace reached the expected project and environment. Storage is asynchronous, so give it a moment.
Check what the trace recorded
Open the trace and confirm the named app root contains the expected model and tool spans, input, output, and session ID.
Read trace health
Read the trace health results. Judgment checks root structure, nesting, timing, input/output, LLM metadata, and session ID consistency. A check with no status is still pending, not passing. See Trace health.
Check for extra traces
Confirm plumbing didn't create extra traces and each request produced one expected trace.
Test a hard case
Test one hard case: restart, stream abort, worker retry, approval resume, handled tool failure, or serverless cold call.
A coding agent can read the same health results with get_trace_health, get_trace_health_summary, and list_trace_health_failures through the Judgment MCP server.
Troubleshooting
| Symptom | Check |
|---|---|
| No project or traces | Check the startup warnings, credentials, organization, key role, network access, and whether real traced code ran. A viewer-role key, a network error, or a missing credential disables export without an exception. |
| Project exists but is empty | Check that the process handling real traffic received the tracing settings and initialization code. |
| Traces reached the wrong project | The name in code doesn't match the project you're looking at, so the SDK created a new one. Fix the name, and log the active project before the root starts. |
Export rejected with 413 | A span has more than 8 MiB of attributes, and the whole batch was rejected. See Ingest contract. |
Export rejected with 402 | The organization hit its usage limit. |
| Root has zero duration or no output | End the root before flushing or stopping the worker. |
| Named root has no integration children or session | Check initialization order and whether the root was active. |
| One request breaks into roots across services | Propagate OpenTelemetry context between services. See Distributed services. |
| Model spans or cost appear twice | Choose manual or automatic LLM instrumentation, not both. |
| Returned failures stay green | Mark handled failures explicitly. See Mark handled failures. |
| Test traces appear with production traffic | Set the SDK's environment option for development and test runs within the same project. See Environment. |
Next step
Add context once the trace structure is healthy. Tracing patterns covers streaming, queues, serverless, and distributed services. The Tracer SDK Reference contains the complete Python and TypeScript APIs.