Skip to Content
Judgment Labs
DocumentationTracingIntegrations

Trace model clients

Trace direct calls to a model provider with wrap() or an OpenTelemetry instrumentor.

If a framework (LangGraph, OpenAI Agents SDK, Claude Agent SDK, Vercel AI SDK) makes model calls for you, use that framework's page instead, so each call is recorded once.

This page assumes your client already makes calls with its own provider credentials. See Set up tracing for Judgment credentials and SDK installation.

wrap() a supported client

wrap() patches the client in place. Every call becomes an LLM span with the model name, token counts, and cost.

ClientPython wrap()TypeScript wrap()
OpenAI (OpenAI, AsyncOpenAI)YesYes
Anthropic (Anthropic, AsyncAnthropic)YesNo, use an instrumentation below
Together (Together, AsyncTogether)YesNo
Google GenAI (google.genai.Client)YesNo
pip install judgeval openai
agent.py
from openai import OpenAI
from judgeval import Tracer, wrap

Tracer.init(project_name="lunch-order-agent")
client = wrap(OpenAI())

@Tracer.observe(span_type="agent")
def handle_request(message: str) -> str:
    response = client.chat.completions.create(
        model="gpt-5.2",
        messages=[{"role": "user", "content": message}],
    )
    return response.choices[0].message.content

print(handle_request("order me mediterranean food to feed 30 hungry engineers"))
Tracer.force_flush()

The other Python clients wrap the same way. Install the clients you use before importing them. The combined example below uses:

python -m pip install anthropic google-genai together
from anthropic import Anthropic
from google import genai
from together import Together
from judgeval import Tracer, wrap

Tracer.init(project_name="lunch-order-agent")

anthropic_client = wrap(Anthropic())
together_client = wrap(Together())
gemini_client = wrap(genai.Client())

For Google, wrap google.genai.Client. The older google.generativeai package (genai.GenerativeModel(...)) isn't detected: wrap() logs "Unknown client type", tries to wrap it as an OpenAI client, and raises TypeError because it isn't one.

OpenAI-compatible endpoints

OpenRouter, Ollama's OpenAI-compatible server, vLLM, and similar endpoints work with wrap(OpenAI(base_url=...)), because the client is still an OpenAI instance. For OpenRouter, use base_url="https://openrouter.ai/api/v1" and pass extra_body={"usage": {"include": True}} on each call so the response carries the cost (OpenRouter's usage accounting).

Other providers: register an instrumentor

For clients wrap() doesn't support, register an OpenTelemetry instrumentor with Judgment. Tracer.registerOTELInstrumentation() calls the instrumentor with Judgment's tracer provider, so the spans go to your project.

from judgeval import Tracer
from opentelemetry.instrumentation.bedrock import BedrockInstrumentor

Tracer.init(project_name="lunch-order-agent")
Tracer.registerOTELInstrumentation(BedrockInstrumentor())
ProviderPython packageImport
AWS Bedrockopentelemetry-instrumentation-bedrockfrom opentelemetry.instrumentation.bedrock import BedrockInstrumentor
Vertex AIopentelemetry-instrumentation-vertexaifrom opentelemetry.instrumentation.vertexai import VertexAIInstrumentor
Groqopentelemetry-instrumentation-groqfrom opentelemetry.instrumentation.groq import GroqInstrumentor
Ollamaopentelemetry-instrumentation-ollamafrom opentelemetry.instrumentation.ollama import OllamaInstrumentor
Mistralopentelemetry-instrumentation-mistralaifrom opentelemetry.instrumentation.mistralai import MistralAiInstrumentor
Hugging Face Transformers (local)opentelemetry-instrumentation-transformersfrom opentelemetry.instrumentation.transformers import TransformersInstrumentor
LiteLLMopentelemetry-instrumentation-litellmfrom opentelemetry.instrumentation.litellm import LiteLLMInstrumentor
Together (instead of wrap())openinference-instrumentation-togetherfrom openinference.instrumentation.together import TogetherInstrumentor

These packages come from OpenLLMetry (opentelemetry-instrumentation-*) and OpenInference (openinference-instrumentation-*). Judgment didn't write them. See OpenTelemetry instrumentation libraries for how each family maps into Judgment.

LiteLLM has no client object to wrap, so the instrumentor is the only option. When sub-agents each call litellm.completion inside their own @Tracer.observe(span_type="agent") function, the cost lands on the model spans under each sub-agent, not on the orchestrator span above them.

In TypeScript, wrap() covers OpenAI only. Passing it another client doesn't throw: the SDK logs an error and returns the client unchanged, so its calls aren't traced. For Anthropic, register new AnthropicInstrumentation() from @traceloop/instrumentation-anthropic with Tracer.registerOTELInstrumentation() in a file your app loads first. The TypeScript tab on OpenTelemetry instrumentation libraries shows the pattern. For Claude Managed Agents sessions, use wrapAnthropicManagedAgents(); see Claude Managed Agents.

If something looks wrong

  • If you see two LLM spans per call, you wrapped the client and also registered an instrumentor (or a framework integration) for it. Keep one.
  • In Python, "Unknown client type" in the logs, then a TypeError, means the client isn't one of the four detected types. Use an instrumentor from the table.
  • In TypeScript, an error logged from wrapOpenAI with no LLM spans means the client isn't an OpenAI client. Use an instrumentation instead.
  • If spans have model and tokens but no input or output, the instrumentor writes prompts under keys Judgment doesn't map yet. Add a mapper; see Attribute keys and mappers.

On this page