Set up Judgment with your coding agent

Paste one prompt into your coding agent, and it connects Judgment, instruments your repo and shows you what Judgment found.

Your coding agent can set up Judgment in your repo from start to finish. Paste this prompt into Claude Code, Codex or Cursor from the root of the repo:

Set up Judgment in this repo by following https://docs.judgmentlabs.ai/documentation/start/agent-setup.md

You sign in once in the browser, and you can create your account there. Claude Code and Codex also need one restart to load the Judgment tools, and the agent tells you how. The agent does the rest: it creates a project and an ingest API key, adds tracing to your agent, sends sample traces, and reports the judges and Issues Judgment created from them.

The rest of this page is the agent’s instructions.

Instructions for the coding agent

Work through these steps in order. Tell the user what you’re doing at each step in one line. Never print, log or commit an API key.

1. Connect the Judgment MCP server

If you already have Judgment MCP tools such as list_organizations, skip to step 2. If the judgment server is already configured but its tools aren’t available, don’t add it again. In Codex, run codex mcp login judgment. Then go to the sign-in table below.

Otherwise, add the server https://mcp.judgmentlabs.ai for the client you’re running in:

In Claude Code and Codex, a server added during a session usually isn’t available until a new session starts. Stop here until the user confirms. Ask them to sign in and restart as their client needs:

Codex runs codex mcp login outside its sandbox, so it asks the user to approve the command first. The continue prompt picks up where this session stopped:

Continue setting up Judgment by following https://docs.judgmentlabs.ai/documentation/start/agent-setup.md

Tell the user what to expect in the browser:

• A new user can create an account with Google or GitHub, then name their organization before authorizing.
• If the page asks for a project, leave All Projects in This Organization selected. A project-bound connection can’t create projects or keys.

After the user confirms, call list_organizations. If the tools still aren’t available, ask the user to start a new session in this directory and paste the continue prompt.

2. Choose the organization and project

1. Call list_organizations. If there’s more than one, ask the user which to use.
2. Call list_projects for that organization. If a project matches this repo’s name, ask whether to reuse it.
3. Otherwise, call create_project with the repo’s directory name.

3. Create an ingest API key

If the repo’s .env already sets JUDGMENT_API_KEY and JUDGMENT_ORG_ID for the organization from step 2, reuse them and skip to step 4. Each key you create stays active until someone deletes it.

Otherwise, call create_ingest_api_key with the repo’s directory name as name. If the name is taken, add a numeric suffix, such as -2.

Write the key and the organization ID to the repo’s .env file, creating it if needed:

JUDGMENT_API_KEY=<api_key from the tool result>
JUDGMENT_ORG_ID=<organization_id from the tool result>

Before you write the key, run git ls-files .env. If it prints .env, the file is tracked by git, so stop and ask the user where to store the key. Otherwise, make sure .gitignore lists .env. The key is shown only once, so don’t call the tool again to read it.

4. Add tracing

An earlier session may have added part of the tracing. Check what’s already there, keep it, and finish only what’s missing.

1. Call read_doc_page for /documentation/connect/set-up-tracing, or fetch https://docs.judgmentlabs.ai/documentation/connect/set-up-tracing.md.
2. Find the agent’s entry points and its model and tool calls. Install the judgeval SDK for the repo’s language.
3. Initialize the tracer once, with the project name from step 2, and load credentials from .env.
4. Trace the agent’s model, tool and agent calls as the tracing page describes. Keep the changes small and leave product behavior as it is.

Show the user the diff before you run anything.

5. Send sample traces

Run the instrumented agent about 10 times with varied, realistic requests. Include a few that should fail safely, such as an out-of-scope request. Use test data or mocks, never production data or real customers.

Flush the tracer before the process exits. Then call get_trace_health_summary for the project. If a check fails, read its remediation, fix the instrumentation, and run the samples again. A pending check isn’t a pass.

6. Show what Judgment found

1. Call start_onboarding_analyses for the project. Judgment creates judges suited to the agent and runs Issue detection over the sample traces. If it returns started: false, its reason says why.
2. Call get_onboarding_status every minute or so until neither run is running or waiting. Tell the user you’re waiting on Judgment.
3. If a run’s status is failed, report its error_message to the user and stop.
4. Call list_judges and list_issues, then summarize for the user:
   • the judges Judgment created and the labels each one applies
   • each Issue found, with its title and what it means for their agent
   • anything you couldn’t verify, such as a failing health check

Link to pages with https://app.judgmentlabs.ai/org/<organization_id>/project/<project_id>/. Add judges/<judge_id> for a judge. End by telling the user that new traces get labeled as they arrive.

If something goes wrong

To set up a repo by hand instead, follow Set up tracing.