Connect your coding agent

Connect your coding agent to Judgment over MCP, then have it set up Judgment in your repo from the docs.

Over MCP, your coding agent can read Issues, traces, judge labels and test runs, then read the code behind a failure and propose the fix in the same session. Handing off an Issue with Fix with relies on it: the prompt tells your agent to read the Issue and its traces through the Judgment MCP server.

The server is https://mcp.judgmentlabs.ai. Every tool is listed in MCP tools. It includes search_docs and read_doc_page, so the same connection can retrieve current product and SDK guidance from these docs.

1. Connect the Judgment MCP server

You need a Judgment account with access to at least one organization and project, and a client with remote HTTP MCP support, such as Cursor, Claude Code or Codex.

Sign in with OAuth

Use browser-based OAuth when your client supports it. The client stores the resulting access token; do not add an API key to its configuration. During authorization, choose the intended organization and leave Project set to All Projects in This Organization. A project-specific grant intentionally hides cross-project discovery tools.

{
  "mcpServers": {
    "judgment": {
      "url": "https://mcp.judgmentlabs.ai"
    }
  }
}

claude mcp add --transport http judgment https://mcp.judgmentlabs.ai

codex mcp add judgment --url https://mcp.judgmentlabs.ai
codex mcp login judgment

[mcp_servers.judgment]
url = "https://mcp.judgmentlabs.ai"

Use an API key only when OAuth is unavailable

Use a personal API key from Settings > Personal API Keys. MCP does not accept ingest API keys. Copy your organization ID with Copy organization ID, and put both in your shell environment:

export JUDGMENT_API_KEY="<your-personal-api-key>"
export JUDGMENT_ORG_ID="<your-organization-id>"

The key has the same permissions as your role. A Viewer’s key can only read; writes need Developer. The client sends the key as Authorization: Bearer <key> and pins the connection to one organization with X-Organization-Id:

[mcp_servers.judgment]
url = "https://mcp.judgmentlabs.ai"
bearer_token_env_var = "JUDGMENT_API_KEY"
env_http_headers = { "X-Organization-Id" = "JUDGMENT_ORG_ID" }

claude mcp add --transport http judgment https://mcp.judgmentlabs.ai \
  --header "Authorization: Bearer $JUDGMENT_API_KEY" \
  --header "X-Organization-Id: $JUDGMENT_ORG_ID"

{
  "mcpServers": {
    "judgment": {
      "url": "https://mcp.judgmentlabs.ai",
      "headers": {
        "Authorization": "Bearer ${env:JUDGMENT_API_KEY}",
        "X-Organization-Id": "${env:JUDGMENT_ORG_ID}"
      }
    }
  }
}

To pin the connection to one project as well, add an X-Project-Id header with the project ID. It requires X-Organization-Id. A project-bound connection hides organization-level tools such as list_projects.

Other clients that support remote HTTP MCP servers use the same URL and headers. Restart or reload your client after changing its MCP config.

Docs MCP server

The documentation has its own read-only MCP server at https://docs.judgmentlabs.ai/mcp. It needs no API key and serves the same pages as this site. Add it the same way if you want docs search without connecting your Judgment data.

2. Check the connection

Ask your agent:

Use the Judgment MCP tools. Call list_organizations, choose the organization
named <organization-name>, then call list_projects for its organization_id.
Return the selected organization_id and the project_id for <project-name>.
Do not make any changes.

Success means the answer contains the real organization_id returned by list_organizations and the real project_id returned by list_projects. If either name is missing, confirm that the signed-in user or API key belongs to the intended organization and that the project exists there. If you get a 401 with an API key, check that JUDGMENT_API_KEY is set in the environment your client runs in.

3. Ask your agent to set up Judgment

Paste this prompt into your coding agent from the root of your repository. Replace the project name and organization ID first. It points the agent at these docs and asks it to set up tracing in your code.

Set up Judgment for project "<project-name>" in organization <organization-id>.

References:
- Judgment docs: https://docs.judgmentlabs.ai (search them with the Judgment MCP
  search_docs and read_doc_page tools, or the docs MCP server at
  https://docs.judgmentlabs.ai/mcp)
- Tracing: https://docs.judgmentlabs.ai/documentation/connect/set-up-tracing
- MCP setup and authentication: https://docs.judgmentlabs.ai/documentation/connect/connect-your-coding-agent
- SDK source: https://github.com/JudgmentLabs/judgeval

1. Read the current docs. Inspect this repository's stack and existing
   instrumentation. Reuse an existing Judgment connection and tracer when
   present; do not create duplicate projects or initialize tracing twice.
2. Verify the coding agent's Judgment MCP connection. If missing, help the user
   add https://mcp.judgmentlabs.ai and authenticate using the documented OAuth
   flow. Use an API key only if OAuth is unavailable. Never put credentials in
   prompts, logs, or tracked files. If the client needs a restart or reload,
   tell the user exactly what to restart, then resume by verifying tools and
   access to this organization and project.
3. Identify the appropriate SDK and entry points, explain the instrumentation
   changes, and follow this repository's approval requirements.
4. Configure the SDK's JUDGMENT_API_KEY and JUDGMENT_ORG_ID securely in the
   user's environment. Use an organization ingest API key (starts with jli_)
   for JUDGMENT_API_KEY. OAuth for MCP does not replace SDK credentials.
   Initialize the tracer once for project "<project-name>" and instrument model,
   tool, and agent calls using documented input/output semantics.
5. Exercise the real instrumented agent entry points with a few varied,
   realistic requests, using mocks or a safe test environment rather than
   production data. Flush the exporter and use the MCP tools to verify that the
   traces, meaningful inputs/outputs, and errors arrived in this project.
6. Check instrumentation health with get_trace_health_summary,
   get_trace_health, and list_trace_health_failures. Read each check's
   remediation, fix instrumentation problems, and recheck. A pending result is
   not a pass. Finish with the traces received, health results, and any
   unresolved problems; do not claim setup is verified from ingestion alone.

Keep instrumentation changes small. Do not alter product behavior, create
judges, configure schedules, apply database migrations, or deploy anything as
part of tracing setup.

The project setup in the app gives you the same kind of prompt, filled in with your project. See Quickstart.

4. Give it a real task

Three tasks that work well on day one:

List the open Issues in lunch-order-agent, pick the one with the highest
severity, read its cited traces, and propose a code fix in this repo.

Check trace health for lunch-order-agent over the last 7 days. For any failing
check, show an example trace and the instrumentation change that would fix it.

Call discover_schema, then use sql_query to find the 10 slowest production
traces this week and the span that took the longest in each.

The first uses list_issues and get_issue, the second get_trace_health_summary and list_trace_health_failures, and the third the SQL tools described in Query with SQL.

If you’d rather ask inside the app, the Judgment Agent uses the same data. To hand a chat from there to your coding agent, use Copy as prompt.