OpenLLMetry Integration

OpenLLMetry integration sends traces from your OpenLLMetry-instrumented applications to Judgment. OpenLLMetry provides OpenTelemetry-based instrumentation for various LLM providers and frameworks.

Quickstart

Install Dependencies

uv add opentelemetry-instrumentation-openai judgeval openai

pip install opentelemetry-instrumentation-openai judgeval openai

Initialize Integration

from judgeval.tracer import Tracer
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

tracer = Tracer(project_name="openllmetry_project")
OpenAIInstrumentor().instrument()

Always initialize the Tracer before calling OpenAIInstrumentor().instrument() to ensure proper trace routing.

Use with OpenAI

from judgeval.tracer import Tracer
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import OpenAI

tracer = Tracer(project_name="openllmetry_project")
OpenAIInstrumentor().instrument()

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

print(response.choices[0].message.content)

All OpenAI calls are automatically traced and exported to the Judgment platform.

Available Instrumentors

OpenLLMetry provides instrumentors for various frameworks and libraries:

OpenAI

from opentelemetry.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument()

Anthropic

from opentelemetry.instrumentation.anthropic import AnthropicInstrumentor
AnthropicInstrumentor().instrument()

OpenAI Agents SDK

from opentelemetry.instrumentation.openai_agents import OpenAIAgentsInstrumentor
OpenAIAgentsInstrumentor().instrument()

Google Generative AI

from opentelemetry.instrumentation.google_generativeai import GoogleGenerativeAiInstrumentor
GoogleGenerativeAiInstrumentor().instrument()

Together AI

from opentelemetry.instrumentation.together import TogetherInstrumentor
TogetherInstrumentor().instrument()

Groq

from opentelemetry.instrumentation.groq import GroqInstrumentor
GroqInstrumentor().instrument()

HuggingFace Transformers

from opentelemetry.instrumentation.transformers import TransformersInstrumentor
TransformersInstrumentor().instrument()

Vertex AI

from opentelemetry.instrumentation.vertexai import VertexAIInstrumentor
VertexAIInstrumentor().instrument()

Ollama

from opentelemetry.instrumentation.ollama import OllamaInstrumentor
OllamaInstrumentor().instrument()

Other

We support the complete list of available instrumentors, so visit the OpenLLMetry GitHub repository for more.

Next Steps

All Integrations - View the list of all integrations that Judgment supports.
Agent Behavior Monitoring - Monitor your AI applications in production with behavioral scoring.
Tracing Reference - Learn more about Judgment's tracing capabilities and advanced configuration.

OpenLLMetry Integration

On this page