MCP tools
Every tool on the Judgment MCP server, with its scope, minimum role and arguments.
The Judgment MCP server gives coding agents and other MCP clients read and write access to one organization’s projects. To set it up in Codex, Claude Code, or Cursor, see Connect your coding agent.
Endpoint
| Item | Value |
|---|---|
| URL | https://mcp.judgmentlabs.ai |
| Transport | Streamable HTTP, stateless. POST only; other methods return 405. |
| Auth | Authorization: Bearer <token>. The token is an OAuth access token or a Judgment API key. A request without a bearer token gets an OAuth challenge. |
| OAuth | OAuth 2.1 with PKCE (S256), issued by https://auth.judgmentlabs.ai. |
| Health | GET /health |
Headers
| Header | Required | Value |
|---|---|---|
Authorization |
Yes | Bearer <oauth-token-or-api-key> |
X-Organization-Id |
No | Organization ID. On an API-key connection, binds the connection to one organization, so tools don’t need organization_id. An OAuth connection is already bound to the organization you chose when you authorized. |
X-Project-Id |
No | Project ID (UUID). Binds the connection to one project. On an API-key connection, send it with X-Organization-Id. |
A malformed X-Project-Id returns 400. X-Project-Id without X-Organization-Id on an API-key connection returns 400. On an OAuth connection, an X-Organization-Id or X-Project-Id that differs from the authorized organization or project returns 400.
Scope and roles
Each tool has a scope and a minimum organization role.
| Scope | Meaning |
|---|---|
| Account | Works for any signed-in user. |
| Organization | Takes organization_id, unless the connection is bound to an organization. |
| Project | Takes organization_id and project_id, unless the connection is bound. Bound fields are removed from the tool’s input schema and filled in by the server. |
Roles from lowest to highest are Viewer, Developer, Admin, and Owner. Viewers can call read tools. Writes need Developer. Deleting a dataset or test config needs Admin. On an OAuth connection, tools above your role are not listed. See Members and roles.
A connection bound to one project (by X-Project-Id or a project-scoped OAuth grant) hides organization and account tools, except list_organizations, discover_schema, search_docs, and read_doc_page.
Some API and SQL names, such as the behavior_id argument, still say behavior; it means label.
Tools
The arguments column lists each tool’s own inputs; ? marks an optional argument. Organization-scoped tools also accept organization_id?, and project-scoped tools accept organization_id? and project_id?; these are not repeated in the tables. evaluate_traces and evaluate_all_traces are enabled per organization; contact support to turn them on. They are listed only on a connection bound to that organization.
New judges are categorical: create_judge takes judgeType prompt (Agent judge), custom (Code judge), or deterministic, and at least two outputs (exactly two for deterministic, match first, then miss). Use get_judge_label and update_judge_label for labels; get_judge_result, get_behavior_detail, update_judge_result, and update_behavior still work as deprecated aliases.
Organizations
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
list_organizations |
Account | Any user | limit?, offset? |
Organizations you belong to. |
Projects
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
create_project |
Organization | Developer | name |
Create a project. |
list_projects |
Organization | Viewer | none | Projects in the organization, with summary counts. |
SQL
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
discover_schema |
Organization | Viewer | none | Markdown reference for the SQL tables, columns, examples, and limits. Call before sql_query. |
sql_query |
Project | Viewer | sql |
Run one read-only SELECT against the project. Up to 1,000 rows and 5 MiB. |
Trace health and evaluation
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
evaluate_all_traces |
Project | Developer | judge_names?, limit? |
Queue judges to run on the most recent traces (default 100, max 1,000). Enabled per organization; contact support to turn it on. |
evaluate_traces |
Project | Developer | trace_ids, judge_names? |
Queue judges to run on specific traces. Enabled per organization; contact support to turn it on. |
get_trace_health |
Project | Viewer | trace_id, category? |
Check results for one trace. |
get_trace_health_summary |
Project | Viewer | category?, start_time, end_time |
Check status and failure rates for a time range of up to 31 days. |
list_trace_health_failures |
Project | Viewer | category?, check_id, start_time, end_time, limit?, cursor_trace_started_at?, cursor_trace_id? |
Traces failing one check in a time range. |
Issues
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
create_issue |
Project | Developer | issue_key?, title, diagnosis, impact_summary?, priority?, production_activity?, agent_reasoning?, cited_items |
Create an Issue backed by cited evidence. A duplicate issue_key returns a conflict. |
get_issue |
Project | Viewer | issue_id |
One Issue with its cited evidence and event history. |
list_issues |
Project | Viewer | user_status?, production_activity?, priority? |
Issues in the project, filterable by review status, activity, and priority. |
update_issue |
Project | Developer | title?, diagnosis?, impact_summary?, priority?, production_activity?, agent_reasoning?, cited_items?, issue_id, user_status?, judge_id?, behavior_id?, message? |
Edit an Issue, comment on it, or change its review status (needs_review, ignored, merged, monitored). resolved is set by the system. |
Judges
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
create_judge |
Project | Developer | judge |
Create an Agent judge, a Code judge (uploaded bundle), or a deterministic judge, with its first production version. The judge type can’t change later. |
delete_judge |
Project | Developer | judge_id |
Delete a judge and the labels that belong to it. |
get_judge |
Project | Viewer | judge_id, include_code_url?, version? |
One version of a judge (latest by default), with its definition, outputs, and settings. For a Code judge, include_code_url returns a temporary download link for its bundle. |
list_judges |
Project | Viewer | version? |
Judges in the project (latest version, or version: "production" for production judges and their enabled labels). |
update_judge |
Project | Developer | judge_id, update |
Edit the description, or create a new version and optionally promote it to production. |
list_judge_action_groups |
Project | Viewer | version? |
Action groups owned by judges, with group IDs and the judge version they belong to (production by default, or latest). |
set_judge_action_groups |
Project | Developer | judge_id, source_version, target_version, promote_to_production, action_groups |
Replace a judge’s action groups by creating a new judge version. Pass the complete list; omitted groups are removed. Call list_judge_action_groups first. |
Labels
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
get_judge_label |
Project | Viewer | behavior_id, start_date?, end_date? |
Rate, last-applied time, and time series for one label on a production judge. Aliases: get_judge_result, get_behavior_detail. |
update_judge_label |
Project | Developer | behavior_id, description? |
Edit a label description. Aliases: update_judge_result, update_behavior. |
Datasets
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
add_examples_to_dataset |
Project | Developer | dataset_id, examples |
Add examples. Creates a new dataset version. |
create_dataset |
Project | Developer | name, schema, examples? |
Create a dataset with a JSON Schema and optional examples. |
delete_dataset |
Project | Admin | dataset_id |
Delete a dataset. Also deletes its test configs and run metadata. |
get_dataset |
Project | Viewer | dataset_id?, name?, version?, limit?, cursor_created_at?, cursor_example_id? |
One dataset (by ID or name) and a page of its examples. |
list_datasets |
Project | Viewer | none | Datasets with schema, example count, and current version. |
remove_examples_from_dataset |
Project | Developer | dataset_id, example_ids |
Remove examples. Creates a new dataset version. |
Test suites
Case edits land in the suite’s draft, which update_test_case and remove_test_case open automatically. publish_test_suite freezes the draft as a version that runs use.
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
add_test_cases |
Project | Developer | test_suite_id, cases |
Add cases to the draft: an input for the agent plus what each judge should return. |
create_test_suite |
Project | Developer | name, description?, labels? |
Create an empty test suite. |
delete_test_suite |
Project | Developer | test_suite_id |
Delete a test suite with its cases, versions, and runs. |
get_test_suite_run_results |
Project | Viewer | run_id |
Run progress and each case’s checks, judge outputs, and pending checks. |
list_test_cases |
Project | Viewer | test_suite_id, version?, offset?, limit? |
Cases at a published version or the draft. |
list_test_suite_runs |
Project | Viewer | test_suite_id?, offset?, limit? |
Runs newest first, with pass and fail counts. |
list_test_suites |
Project | Viewer | offset?, limit? |
Test suites with draft state and recent runs. |
publish_test_suite |
Project | Developer | test_suite_id |
Publish the draft as a new version. |
remove_test_case |
Project | Developer | test_suite_id, case_id |
Remove a case from the draft. |
start_test_suite_run |
Project | Developer | test_suite_id, case_filter?, candidate?, candidate_meta? |
Start a run and get each case’s input and trace headers. |
update_test_case |
Project | Developer | test_suite_id, case_id, name?, input?, expected?, tags?, metadata? |
Change fields of a case in the draft. |
Tests
These dataset test tools are listed only for organizations with legacy tests enabled.
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
cancel_test_run |
Project | Developer | test_run_id |
Cancel a pending or running test run. |
create_test_config |
Project | Developer | name, description?, dataset_id, judge_ids |
Pair a dataset with judges. |
delete_test_config |
Project | Admin | test_config_id |
Delete a test config and its run metadata. |
get_test_run |
Project | Viewer | test_run_id |
Status and metadata for one test run. |
get_test_run_items |
Project | Viewer | test_run_id, limit?, cursor? |
Per-example results and scores for one test run. |
list_test_configs |
Project | Viewer | dataset_id? |
Test configs (a dataset paired with judges). |
list_test_runs |
Project | Viewer | test_config_id?, dataset_id?, status?, limit? |
Test runs, newest first. |
run_test |
Project | Developer | test_config_id, dataset_version_number?, judge_versions? |
Queue a test run that scores every example with every judge in the config. |
update_test_config |
Project | Developer | test_config_id, name?, description? |
Rename a test config. The dataset and judges are fixed. |
Scheduled tasks
A scheduled task (formerly called a scheduled agent) is a custom agent. These tools use the same permissions as the Scheduled page: anyone with Developer access can create a task, and only its author or an organization admin can edit it.
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
create_scheduled_task |
Project | Developer | name, description?, instructions?, mode?, tool_permission_mode?, schedule?, slack_channel_ids? |
Create a scheduled task. |
get_scheduled_task |
Project | Viewer | task_id |
One scheduled task with its instructions, schedule, and Slack delivery. |
list_scheduled_tasks |
Project | Viewer | none | Scheduled tasks in the project. |
update_scheduled_task |
Project | Developer | task_id, name?, description?, instructions?, mode?, tool_permission_mode?, schedule?, slack_channel_ids? |
Edit a scheduled task. Send at least one field. An empty slack_channel_ids stops Slack delivery. |
mode is fast or deep_research. tool_permission_mode is auto_require_permission (ask before write actions) or always_accept. schedule is { enabled, cadence?, timezone? }; leaving out cadence keeps the saved one, so { "enabled": false } pauses a task. A cadence is a daily or weekly time (frequency, hour, minute, and dayOfWeek or daysOfWeek), an interval (mode: "interval", every, and unit of minutes from 15 to 1440 or hours from 1 to 24), or a five-field cron expression (mode: "cron", cronExpression).
Agent Knowledge
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
list_memory_entries |
Project | Viewer | none | Paths of Agent Knowledge files and folders. |
read_memory_files |
Project | Viewer | ids?, names? |
Read Agent Knowledge files by ID or path. |
search_memory_files |
Project | Viewer | query, limit? |
Search Agent Knowledge files by path and body. |
write_memory_file |
Project | Developer | name, body |
Create or update one Agent Knowledge file by path. |
Attribute mappers
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
create_mapper |
Project | Developer | name, rules, enabled?, override?, remove_source? |
Create an attribute mapper. Org-wide mappers need a Judgment admin. |
delete_mapper |
Project | Developer | config_id |
Delete an attribute mapper. |
get_mapper |
Project | Viewer | config_id |
One attribute mapper. |
list_mappers |
Project | Viewer | none | Attribute mappers for the project, including org-wide mappers. |
update_mapper |
Project | Developer | config_id, name?, rules?, enabled?, override?, remove_source? |
Edit an attribute mapper. |
Docs
| Tool | Scope | Minimum role | Arguments | What it does |
|---|---|---|---|---|
read_doc_page |
Account | Any user | route |
Read one Judgment docs page as Markdown. |
search_docs |
Account | Any user | query, contentTypes?, filters?, limit? |
Search the Judgment docs. |
Retired tools
These tools were removed in September 2026. Calling one returns Tool <name> not found.
| Retired tool | Use instead |
|---|---|
search_traces, get_trace_detail, get_trace_spans, get_trace_span, read_span_attribute, get_trace_tags, get_trace_behaviors |
sql_query on telemetry.traces, telemetry.spans, telemetry.scores, telemetry.trace_tags |
search_sessions, get_session_detail, get_session_trace_ids, get_session_behaviors, get_session_trace_behaviors |
sql_query, grouping telemetry.traces by session_id |
get_test_run_graph |
sql_query on evaluations.items |
list_agent_memory_entries, fetch_agent_memory_files, search_agent_memory_files, write_memory |
list_memory_entries, read_memory_files, search_memory_files, write_memory_file |
write_issues, ignore_issue, reopen_issue, write_issue_comment, monitor_issue_with_judge |
create_issue, update_issue |
delete_judges |
delete_judge |
For the SQL tables, see SQL schema.