--- title: "Quickstart" description: "Instrument one real agent path, send a trace to Judgment, apply an Agent Judge, and inspect the result." sidebar: label: "Quickstart" seo: title: "Trace and Evaluate Your First AI Agent | Judgment Tutorial" description: "Install Judgeval, trace a real AI agent request, apply an Agent Judge, and inspect the evaluation result in Judgment." --- After this quickstart, Judgment will show a real agent trace and its Agent Judge result. The example uses OpenAI, but the same flow applies to an existing agent. {/* @test-flow id: quickstart-python lang: python env: JUDGMENT_API_KEY, JUDGMENT_ORG_ID, JUDGMENT_PROJECT_NAME, OPENAI_API_KEY */} {/* @test-flow id: quickstart-typescript lang: typescript env: JUDGMENT_API_KEY, JUDGMENT_ORG_ID, JUDGMENT_PROJECT_NAME, OPENAI_API_KEY */} ## Prerequisites - A Judgment account with a project - Developer, admin, or owner access to the project's organization - A Judgment API key from **Settings > Account & API Key** - The organization ID and exact project name shown in Judgment - An OpenAI API key for the example, or an existing agent path you can run > **Using a framework or another model provider** > > Keep this tutorial's outcome and follow the matching > [integration guide](/documentation/integrations) for the > instrumentation code. Do not stack manual model instrumentation on top of an > integration that already records the same call. > **Optional: connect your coding agent** > > The [Judgment MCP server](/documentation/mcp-and-agent-tools) lets Cursor, > Claude Code, Windsurf, Codex, and other coding agents inspect the same > project while you work through this tutorial. Connect it now or after your > first successful trace; it is not required for the steps below. 1. ## Install and configure Judgeval Install Judgeval and the OpenAI client used by the example. The TypeScript path also installs `tsx` to run the `.ts` file directly. **Python** ```bash pip install judgeval openai python-dotenv ``` **TypeScript** ```bash npm install judgeval openai npm install --save-dev tsx ``` Set the credentials and project name in the environment that will run your agent. Keep the API keys out of source control. ```dotenv title=".env" JUDGMENT_API_KEY="your_judgment_api_key" JUDGMENT_ORG_ID="your_organization_id" JUDGMENT_PROJECT_NAME="your_exact_project_name" OPENAI_API_KEY="your_openai_api_key" ``` The API key authenticates the process, the organization ID selects the account, and the project name routes the trace. A typo in the project name can create or target a different project, so copy it exactly. 2. ## Instrument one real agent path Initialize tracing before you create wrapped clients or observed functions. Then put one `agent` root around the user-visible work and let model and tool spans become its children. **Python** ```python title="trace_agent.py" import os from dotenv import load_dotenv from judgeval import Tracer from openai import OpenAI load_dotenv() Tracer.init(project_name=os.environ["JUDGMENT_PROJECT_NAME"]) openai = Tracer.wrap(OpenAI()) @Tracer.observe(span_type="tool", span_name="retrieve_context") def retrieve_context(question: str) -> str: return f"Reference material relevant to: {question}" @Tracer.observe(span_type="agent", span_name="run_agent") def run_agent(question: str) -> str: context = retrieve_context(question) response = openai.chat.completions.create( model="gpt-5.2", messages=[{ "role": "user", "content": f"Question: {question}\n\nContext: {context}", }], ) return response.choices[0].message.content if __name__ == "__main__": print(run_agent("What is the capital of the United States?")) Tracer.shutdown() ``` Run the real path: ```bash python trace_agent.py ``` **TypeScript** ```typescript title="traceAgent.ts" import { Tracer } from "judgeval"; import OpenAI from "openai"; await Tracer.init({ projectName: process.env.JUDGMENT_PROJECT_NAME, }); const openai = Tracer.wrap(new OpenAI({ apiKey: process.env.OPENAI_API_KEY, })); const retrieveContext = Tracer.observe(async function retrieveContext( question: string, ): Promise { return `Reference material relevant to: ${question}`; }, { spanType: "tool", spanName: "retrieve_context" }); const runAgent = Tracer.observe(async function runAgent( question: string, ): Promise { const context = await retrieveContext(question); const response = await openai.chat.completions.create({ model: "gpt-5.2", messages: [{ role: "user", content: `Question: ${question}\n\nContext: ${context}`, }], }); return response.choices[0]?.message.content ?? "No answer"; }, { spanType: "agent", spanName: "run_agent" }); console.log( await runAgent("What is the capital of the United States?"), ); await Tracer.shutdown(); ``` Run the real path: ```bash npx tsx --env-file=.env traceAgent.ts ``` For an existing app, instrument its real request, job, or agent entry point instead of keeping this sample. The full [Tracing guide](/documentation/tracing) covers streaming, durable work, serverless runtimes, distributed systems, and manual OpenTelemetry. 3. ## Verify the trace In your Judgment project, open **Logs > Traces**, then open the newest `run_agent` trace. Confirm that it shows: - One `run_agent` root with the question as input and the answer as output - A `retrieve_context` tool span and one model span under that root - A completed duration rather than an open or zero-duration root - Trace health results with no unexplained failures > **Initialization is not proof** > > A successful `Tracer.init()` only proves that setup ran. The trace in the correct > Judgment project is the proof that the agent executed traced code and exported > it successfully. > **Use Judgment Agent on this trace** > > With the trace open, ask Judgment Agent: `Summarize what this agent did and > point me to any suspicious span.` The current trace and focused span attach > automatically, so its answer should cite the evidence already on the page. 4. ## Create and monitor a simple Agent Judge 1. Open **Judges** and select **New Judge**. 2. Choose **Create manually**, then choose **Binary**. 3. Name the judge `Uses retrieved context`. 4. Keep the default reasoning level and use this prompt: ```text Return true when the agent retrieves context before answering the user. Return false when it answers without retrieving context. ``` 5. Select **Create Judge**. 6. In the judge's **Details** panel, select **Set up monitoring**. 7. In **Advanced Settings**, keep **Continuous** evaluation, use `100%` sampling, keep trace scoring, and leave span triggers empty. 8. Select **Update**. This judge returns a true-or-false result for the evidence in each trace. The example's `retrieve_context` span gives it an unambiguous first case. Its binary output is now a behavior that Judgment evaluates on new traces. 5. ## Run one evaluated trace Run the same agent path again after monitoring is enabled. You do not need to add an evaluation call to the application; Judgment evaluates new traces using the judge's continuous monitoring settings. **Python** ```bash python trace_agent.py ``` **TypeScript** ```bash npx tsx --env-file=.env traceAgent.ts ``` When the new `run_agent` span completes, Judgment queues the judge for that trace. Evaluation is asynchronous, so the result can appear a few seconds after the trace. 6. ## Inspect the judge result Return to **Logs > Traces** and open the newest `run_agent` trace. In the right-side **Behaviors** panel, open the `Uses retrieved context` result. Verify that the result includes: - The judge name and its true-or-false decision - The reasoning for the decision - Evidence that points back to the relevant trace content You now have the complete first-success path: one real agent trace and one inspected judge result visible in Judgment. If the result does not appear, confirm that the judge is set to **Continuous** with `100%` sampling, the traced run completed after monitoring was enabled, and the trace reached the same project as the judge. Then refresh or reopen the trace. ## Continue the loop Run the judge on incoming traces, watch trends, and alert on failures. Add representative evidence to a dataset and compare a proposed change.