Trace model clients

Trace direct calls to a model provider with wrap() or an OpenTelemetry instrumentor.

If a framework (LangGraph, OpenAI Agents SDK, Claude Agent SDK, Vercel AI SDK) makes model calls for you, use that framework’s page instead, so each call is recorded once.

wrap() a supported client

wrap() patches the client in place. Every call becomes an LLM span with the model name, token counts, and cost.

pip install judgeval openai

from openai import OpenAI
from judgeval import Tracer, wrap

Tracer.init(project_name="lunch-order-agent")
client = wrap(OpenAI())

@Tracer.observe(span_type="agent")
def handle_request(message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-5.2",
        messages=[{"role": "user", "content": message}],
    )
    return response.choices[0].message.content

print(handle_request("order me mediterranean food to feed 30 hungry engineers"))
Tracer.force_flush()

npm install judgeval openai

import OpenAI from "openai";
import { Tracer, wrap } from "judgeval";

await Tracer.init({ projectName: "lunch-order-agent" });
const client = wrap(new OpenAI());

const handleRequest = Tracer.observe(
  async function handleRequest(message: string) {
    const response = await client.chat.completions.create({
      model: "gpt-5.2",
      messages: [{ role: "user", content: message }],
    });
    return response.choices[0].message.content;
  },
  { spanType: "agent" },
);

console.log(
  await handleRequest("order me mediterranean food to feed 30 hungry engineers"),
);
await Tracer.shutdown();

The other Python clients wrap the same way:

from anthropic import Anthropic
from google import genai
from together import Together
from judgeval import Tracer, wrap

Tracer.init(project_name="lunch-order-agent")

anthropic_client = wrap(Anthropic())
together_client = wrap(Together())
gemini_client = wrap(genai.Client())

For Google, wrap google.genai.Client. The older google.generativeai package (genai.GenerativeModel(...)) isn’t detected: wrap() logs “Unknown client type”, tries to wrap it as an OpenAI client, and raises TypeError because it isn’t one.

OpenAI-compatible endpoints

OpenRouter, Ollama’s OpenAI-compatible server, vLLM, and similar endpoints work with wrap(OpenAI(base_url=...)), because the client is still an OpenAI instance. For OpenRouter, use base_url="https://openrouter.ai/api/v1" and pass extra_body={"usage": {"include": True}} on each call so the response carries the cost (OpenRouter’s usage accounting).

Other providers: register an instrumentor

For clients wrap() doesn’t support, register an OpenTelemetry instrumentor with Judgment. Tracer.registerOTELInstrumentation() calls the instrumentor with Judgment’s tracer provider, so the spans go to your project.

from judgeval import Tracer
from opentelemetry.instrumentation.bedrock import BedrockInstrumentor

Tracer.init(project_name="lunch-order-agent")
Tracer.registerOTELInstrumentation(BedrockInstrumentor())

These packages come from OpenLLMetry (opentelemetry-instrumentation-*) and OpenInference (openinference-instrumentation-*). Judgment didn’t write them. See OpenTelemetry instrumentation libraries for how each family maps into Judgment.

LiteLLM has no client object to wrap, so the instrumentor is the only option. When sub-agents each call litellm.completion inside their own @Tracer.observe(span_type="agent") function, the cost lands on the model spans under each sub-agent, not on the orchestrator span above them.

In TypeScript, wrap() covers OpenAI only. Passing it another client doesn’t throw: the SDK logs an error and returns the client unchanged, so its calls aren’t traced. For Anthropic, register new AnthropicInstrumentation() from @traceloop/instrumentation-anthropic with Tracer.registerOTELInstrumentation() in a file your app loads first. The TypeScript tab on OpenTelemetry instrumentation libraries shows the pattern. For Claude Managed Agents sessions, use wrapAnthropicManagedAgents(); see Claude Managed Agents.

If something looks wrong

• If you see two LLM spans per call, you wrapped the client and also registered an instrumentor (or a framework integration) for it. Keep one.
• In Python, “Unknown client type” in the logs, then a TypeError, means the client isn’t one of the four detected types. Use an instrumentor from the table.
• In TypeScript, an error logged from wrapOpenAI with no LLM spans means the client isn’t an OpenAI client. Use an instrumentation instead.
• If spans have model and tokens but no input or output, the instrumentor writes prompts under keys Judgment doesn’t map yet. Add a mapper; see Attribute keys and mappers.