Judgeval
Run evaluations and manage tracing with the Judgeval class - your primary interface to the Judgment platform.
The Judgeval class is your primary interface for interacting with the Judgment platform. It provides access to tracing, scoring, and evaluation functionality through a unified API.
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
client_with_config = Judgeval(project_name="my-project", api_key="your_api_key", organization_id="your_organization_id")Authentication
Set up your credentials using environment variables:
export JUDGMENT_API_KEY="your_key_here"
export JUDGMENT_ORG_ID="your_org_id_here" # Add to your .env file
JUDGMENT_API_KEY="your_key_here"
JUDGMENT_ORG_ID="your_org_id_here" Judgeval(project_name="default_project")
Initialize a Judgeval client instance.
from judgeval import Judgeval
client = Judgeval(
project_name: str,
api_key: str | None = None,
organization_id: str | None = None,
api_url: str | None = None,
)Parameters
project_namerequired
:strThe name of the project to bind the client to.
api_key
:str | NoneYour Judgment API key.
Recommended: set with JUDGMENT_API_KEY environment variable.
organization_id
:str | NoneYour organization ID.
Recommended: set with JUDGMENT_ORG_ID environment variable.
api_url
:str | NoneAPI URL for custom endpoints.
Recommended: set with JUDGMENT_API_URL environment variable.
Behavior
Throws
Errorif API key or organization ID are not found
Example
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
client_with_config = Judgeval(project_name="my-project", api_key="your_api_key", organization_id="your_organization_id")Accessing Features
The Judgeval client provides access to various features through factory properties:
client.tracer
Access the tracer factory for creating tracers.
client.tracer.create() : TracerSee Tracer for detailed documentation.
client.scorers
Access scorer factories for creating and retrieving scorers.
client.scorers.builtIn.answerRelevancy()
client.scorers.builtIn.answerCorrectness()
client.scorers.builtIn.faithfulness()
client.scorers.promptScorer.get(name=<prompt scorer name>)
client.scorers.tracePromptScorer.get(name=<prompt scorer name>)
client.scorers.customScorer.get(name=<prompt scorer name>)See PromptScorer for detailed documentation.
client.evaluation
Access the evaluation factory for running batch evaluations.
client.evaluation.create(): EvaluationComplete Usage Example
Here's a complete example showing how to use the new Judgeval API with tracing and evaluation:
import os
from openai import OpenAI
from judgeval import Judgeval
from judgeval.v1.data import Example
client = Judgeval(project_name="customer_support_agent")
tracer = client.tracer.create()
openai = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
# Wrap the OpenAI client for automatic instrumentation
openai = tracer.wrap(openai)
@tracer.observe
def chat_with_user(user_message: str) -> str:
response = openai.chat.completions.create(
model="gpt-5.2",
messages=[{"role": "user", "content": user_message}],
)
result = response.choices[0].message.content or ""
tracer.async_evaluate(
scorer=client.scorers.built_in.answer_relevancy(),
example=Example.create(
input=user_message,
actual_output=result,
),
)
return result
result = chat_with_user("What is the capital of France?")
print(result)