Get Started

Judgeval is an Agent Behavior Monitoring (ABM) library that helps track and judge any agent behavior in online and offline environments. Judgeval also provides error analysis on agent trajectories and groups trajectories by behavior and topic for deeper analysis.

Judgeval is built and maintained by Judgment Labs. You can follow our latest updates via GitHub.

Quickstart

This quickstart will guide you through the core features of Judgeval and the Judgment Platform. By the end, you'll be familiar with the core concepts and be able to start monitoring your agents in production.

Install judgeval

uv add judgeval

pip install judgeval

npm install judgeval

yarn add judgeval

pnpm add judgeval

bun add judgeval

Get your API keys

Head to the Judgment Platform and create an account. Then, copy your API key and Organization ID and set them as environment variables.

Get your free API keys

You get 50,000 free trace spans and 1,000 free evals each month. No credit card required.

.env

JUDGMENT_API_KEY="your_key_here"
JUDGMENT_ORG_ID="your_org_id_here"

Trace your Agent

Tracing captures your agent's inputs, outputs, tool calls, and LLM calls to help you debug and analyze agent behavior.

Note: This example uses OpenAI. Make sure you have OPENAI_API_KEY set in your environment variables before running.

To properly trace your agent, you need to:

Use @tracer.observe() decorator on all functions and tools of your agent
Use tracer.wrap() to instrument all LLM client calls (e.g., tracer.wrap(OpenAI()))

trace_agent.py

from openai import OpenAI
from judgeval import Judgeval
import time

judgeval = Judgeval(project_name="default_project")

tracer = judgeval.tracer.create()  # organizes traces
client = tracer.wrap(OpenAI())  # tracks all LLM calls


@tracer.observe(span_type="tool") 
def format_task(question: str) -> str:
    time.sleep(0.5)  # Simulate some processing delay
    return f"Please answer the following question: {question}"


@tracer.observe(span_type="tool") 
def answer_question(prompt: str) -> str:
    time.sleep(0.3)  # Simulate some processing delay
    response = client.chat.completions.create(
        model="gpt-5.2", messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content


@tracer.observe(span_type="function") 
def run_agent(question: str) -> str:
    task = format_task(question)
    answer = answer_question(task)
    return answer


if __name__ == "__main__":
    result = run_agent("What is the capital of the United States?")
    print(result)

To properly trace your agent, you need to:

Use judgment.observe(...) to wrap all functions and tools of your agent

traceAgent.ts

import { Judgeval } from "judgeval";
import OpenAI from "openai";

const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
});

const judgeval = Judgeval.create();

const tracer = await judgeval.nodeTracer.create({
    projectName: "default_project",
});

const runAgent = tracer.observe(async function runAgent( 
    question: string
): Promise<string> {
    const task = await formatTask(question);
    const answer = await answerQuestion(task);
    return answer;
},
"function");

const formatTask = tracer.observe(async function formatTask( 
    question: string
): Promise<string> {
    await new Promise((resolve) => setTimeout(resolve, 500));
    return `Please answer the following question: ${question}`;
},
"tool");

const answerQuestion = tracer.observe(async function answerQuestion( 
    prompt: string
): Promise<string> {
    await new Promise((resolve) => setTimeout(resolve, 300));
    return await openAICompletion(prompt);
},
"tool");

const openAICompletion = tracer.observe(async function openAICompletion( 
    prompt: string
): Promise<string> {
    const response = await openai.chat.completions.create({
        model: "gpt-5.2",
        messages: [{ role: "user", content: prompt }],
    });
    return response.choices[0]?.message.content || "No answer";
},
"llm");

await runAgent("What is the capital of the United States?");
await tracer.shutdown();

Congratulations! You've just created your first trace. It should look like this:

Create a Behavior Judge

Online behavioral monitoring lets you run judges directly on your agents in production. Engineers can be alerted the instant an agent misbehaves and make proactive fixes before customers are affected.

In Judgment, an LLM Judge is a judge that evaluates a full trace using a natural language rubric. You can create one by first navigating to the Judges section in the sidebar of the Judgment Platform.

Click the New Judge button and select Binary (Behaviors can be associated with binary or classification judges).

Configure your judge:

Name the judge Helpfulness Scorer
Select a judge model (e.g., gpt-5.2)

Set your judge prompt that defines the behavior to evaluate:

Does the agent call relevant tools effectively to help the user with their request?

Finally, click Create Judge to save your judge.

Monitor Your Agent Using LLM Judges

Modify your agent's tracing to asynchronously evaluate behavior with the Helpfulness Scorer.

Add the scorer_config parameter to a @judgment.observe() decorator.

Adding the scorer_config parameter to the @judgment.observe() decorator of the top-level function that invokes your agent will automatically score it.

trace_agent.py

from openai import OpenAI
from judgeval import Judgeval
import time

judgeval = Judgeval(project_name="default_project")

tracer = judgeval.tracer.create()  # organizes traces
client = tracer.wrap(OpenAI())  # tracks all LLM calls

# Retrieve an LLM Judge (PromptScorer) created on the platform
llm_judge = judgeval.scorers.prompt_scorer.get(
    name="Helpfulness Scorer"
)


@tracer.observe(span_type="tool")
def format_task(question: str) -> str:
    time.sleep(0.5)  # Simulate some processing delay
    return f"Please answer the following question: {question}"


@tracer.observe(span_type="tool")
def answer_question(prompt: str) -> str:
    time.sleep(0.3)  # Simulate some processing delay
    response = client.chat.completions.create(
        model="gpt-5.2", messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content


@tracer.observe(span_type="function")
def run_agent(question: str) -> str:
    tracer.async_trace_evaluate(
        scorer=llm_judge,
    )
    task = format_task(question)
    answer = answer_question(task)
    return answer


if __name__ == "__main__":
    result = run_agent("What is the capital of the United States?")
    print(result)

Use the tracer's asyncTraceEvaluate() method within an observed trace span.

traceAgent.ts

import { Judgeval } from "judgeval";
import OpenAI from "openai";

const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
});

const judgeval = Judgeval.create();

const tracer = await judgeval.nodeTracer.create({
    projectName: "default_project",
});

// Retrieve an LLM Judge (PromptScorer) created on the platform
const llmJudge = await judgeval.scorers.promptScorer.get(
    "Helpfulness Scorer"
);

const runAgent = tracer.observe(async function runAgent(
    question: string
): Promise<string> {
    tracer.asyncTraceEvaluate(llmJudge); 

    const task = await formatTask(question);
    const answer = await answerQuestion(task);

    return answer;
},
"function");

const formatTask = tracer.observe(async function formatTask(
    question: string
): Promise<string> {
    await new Promise((resolve) => setTimeout(resolve, 500));
    return `Please answer the following question: ${question}`;
},
"tool");

const answerQuestion = tracer.observe(async function answerQuestion(
    prompt: string
): Promise<string> {
    await new Promise((resolve) => setTimeout(resolve, 300));
    return await openAICompletion(prompt);
},
"tool");

const openAICompletion = tracer.observe(async function openAICompletion(
    prompt: string
): Promise<string> {
    const response = await openai.chat.completions.create({
        model: "gpt-5.2",
        messages: [{ role: "user", content: prompt }],
    });
    return response.choices[0]?.message.content || "No answer";
},
"llm");

await runAgent("What is the capital of the United States?");
await tracer.shutdown();

Now incoming traces will be automatically evaluated with the Helpfulness Scorer. Check out your first agent trace with behavior monitoring in the Judgment Platform:

You can also instrument agents' behavior with judgeval to send alerts when agents are misbehaving in production.

Next Steps

Congratulations! You've just finished getting started with judgeval and the Judgment Platform.

Explore our features in more detail below:

Agent Judges - Measure and optimize your agent along any behavioral rubric, using techniques such as LLM-as-a-judge and human-aligned rubrics.
Agent Behavior Monitoring - Take action when your agents misbehave in production: alert your team, add failure cases to datasets for later optimization, and more.