Quickstart

Install and configure Judgeval

Install Judgeval and the OpenAI client used by the example. The TypeScript path also installs tsx to run the .ts file directly.

pip install judgeval openai python-dotenv

npm install judgeval openai
npm install --save-dev tsx

Set the credentials and project name in the environment that will run your agent. Keep the API keys out of source control.

JUDGMENT_API_KEY="your_judgment_api_key"
JUDGMENT_ORG_ID="your_organization_id"
JUDGMENT_PROJECT_NAME="your_exact_project_name"
OPENAI_API_KEY="your_openai_api_key"

The API key authenticates the process, the organization ID selects the account, and the project name routes the trace. A typo in the project name can create or target a different project, so copy it exactly.

Instrument one real agent path

Initialize tracing before you create wrapped clients or observed functions. Then put one agent root around the user-visible work and let model and tool spans become its children.

import os

from dotenv import load_dotenv
from judgeval import Tracer
from openai import OpenAI


load_dotenv()
Tracer.init(project_name=os.environ["JUDGMENT_PROJECT_NAME"])
openai = Tracer.wrap(OpenAI())


@Tracer.observe(span_type="tool", span_name="retrieve_context")
def retrieve_context(question: str) -> str:
    return f"Reference material relevant to: {question}"


@Tracer.observe(span_type="agent", span_name="run_agent")
def run_agent(question: str) -> str:
    context = retrieve_context(question)
    response = openai.chat.completions.create(
        model="gpt-5.2",
        messages=[{
            "role": "user",
            "content": f"Question: {question}\n\nContext: {context}",
        }],
    )
    return response.choices[0].message.content


if __name__ == "__main__":
    print(run_agent("What is the capital of the United States?"))
    Tracer.shutdown()

python trace_agent.py

import { Tracer } from "judgeval";
import OpenAI from "openai";

await Tracer.init({
  projectName: process.env.JUDGMENT_PROJECT_NAME,
});

const openai = Tracer.wrap(new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
}));

const retrieveContext = Tracer.observe(async function retrieveContext(
  question: string,
): Promise<string> {
  return `Reference material relevant to: ${question}`;
}, { spanType: "tool", spanName: "retrieve_context" });

const runAgent = Tracer.observe(async function runAgent(
  question: string,
): Promise<string> {
  const context = await retrieveContext(question);
  const response = await openai.chat.completions.create({
    model: "gpt-5.2",
    messages: [{
      role: "user",
      content: `Question: ${question}\n\nContext: ${context}`,
    }],
  });
  return response.choices[0]?.message.content ?? "No answer";
}, { spanType: "agent", spanName: "run_agent" });

console.log(
  await runAgent("What is the capital of the United States?"),
);
await Tracer.shutdown();

npx tsx --env-file=.env traceAgent.ts

For an existing app, instrument its real request, job, or agent entry point instead of keeping this sample. The full Tracing guide covers streaming, durable work, serverless runtimes, distributed systems, and manual OpenTelemetry.

Verify the trace

In your Judgment project, open Logs > Traces, then open the newest run_agent trace.

Confirm that it shows:

• One run_agent root with the question as input and the answer as output
• A retrieve_context tool span and one model span under that root
• A completed duration rather than an open or zero-duration root
• Trace health results with no unexplained failures

Create and monitor a simple Agent Judge

1. Open Judges and select New Judge.

2. Choose Create manually, then choose Binary.

3. Name the judge Uses retrieved context.

4. Keep the default reasoning level and use this prompt:

   Return true when the agent retrieves context before answering the user.
   Return false when it answers without retrieving context.

5. Select Create Judge.

6. In the judge’s Details panel, select Set up monitoring.

7. In Advanced Settings, keep Continuous evaluation, use 100% sampling, keep trace scoring, and leave span triggers empty.

8. Select Update.

This judge returns a true-or-false result for the evidence in each trace. The example’s retrieve_context span gives it an unambiguous first case. Its binary output is now a behavior that Judgment evaluates on new traces.

Run one evaluated trace

Run the same agent path again after monitoring is enabled. You do not need to add an evaluation call to the application; Judgment evaluates new traces using the judge’s continuous monitoring settings.

python trace_agent.py

npx tsx --env-file=.env traceAgent.ts

When the new run_agent span completes, Judgment queues the judge for that trace. Evaluation is asynchronous, so the result can appear a few seconds after the trace.

Inspect the judge result

Return to Logs > Traces and open the newest run_agent trace. In the right-side Behaviors panel, open the Uses retrieved context result.

Verify that the result includes:

• The judge name and its true-or-false decision
• The reasoning for the decision
• Evidence that points back to the relevant trace content

You now have the complete first-success path: one real agent trace and one inspected judge result visible in Judgment.

If the result does not appear, confirm that the judge is set to Continuous with 100% sampling, the traced run completed after monitoring was enabled, and the trace reached the same project as the judge. Then refresh or reopen the trace.