Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

MCP tools

Reference for every currently registered Judgment data MCP tool, category, access level, and input field.

The Judgment data MCP groups its tools by product area. The schema returned by your client’s tools/list call is authoritative for the tools and fields available to that connection.

Scope and access

  • An unbound API-key connection supplies organization_id to every organization- or project-scoped tool and project_id to every project-scoped tool.
  • OAuth authorization binds the connection to an organization and optionally a project. The MCP server removes bound scope fields from the model-facing schema and injects them server-side. Project-bound connections also hide account- or organization-level tools that fall outside the selected project.
  • Read tools inspect state. Write · developer and Write · admin require at least that organization role. A plain Write tool has more specific authorization in its description or the underlying resource.
  • The Inputs column lists the registered field names beyond the common scope fields. Use tools/list for which fields are required and their exact nested schemas.

Organizations

Tool Access Inputs Purpose
list_organizations Read limit, offset List organizations available to the authenticated user. An OAuth-scoped connection returns its selected organization.

Projects

These tools are organization-scoped. The favorite tools keep project_id as an explicit mutation target even when the connection is organization-bound.

Tool Access Inputs Purpose
list_projects Read — List projects with dataset, test-run, trace, and behavior summary counts.
create_project Write · developer name Create a project.
add_project_favorite Write project_id Add a project to the current user’s favorites.
remove_project_favorite Write project_id Remove a project from the current user’s favorites.

Traces

Tool Access Inputs Purpose
search_traces Read queries Run up to 20 trace searches in one server-side batch.
get_trace_detail Read trace_id Read trace duration, cost, session, and summary detail.
get_trace_spans Read trace_id Read the trace’s span-tree metadata.
get_trace_span Read spans Batch-read details for up to 20 trace/span pairs.
read_span_attribute Read trace_id, span_id, attribute_key, offset, length Page through one full span attribute value.
get_trace_tags Read trace_id Read trace tags.
get_trace_behaviors Read trace_id Read binary and categorical behavior results.
get_trace_health_summary Read start_time, end_time Read project Trace Health status and failure rates for a window of at most 31 days.
get_trace_health Read trace_id Read every registered Trace Health check for one trace.
list_trace_health_failures Read check_id, start_time, end_time, limit, cursor_trace_started_at, cursor_trace_id Page through traces failing one health check.
add_trace_tags Write · developer trace_id, tags Add tags without replacing existing tags.
evaluate_traces Write · developer trace_ids, judge_names Evaluate up to 100 named traces, optionally with selected judges.
evaluate_all_traces Write · developer judge_names, limit Evaluate the most recent traces across all time, up to 1,000.
get_trace_attribute_columns Read — Read the shared custom root-span columns on the default traces view.
set_trace_attribute_columns Write · developer attribute_keys Replace the complete shared attribute-column list, up to 20 keys.

search_traces query shape

queries is an array of one to 20 objects. Each query accepts:

Field Shape
id Optional caller ID echoed in the corresponding result.
filters Optional array of filter objects.
time_range Optional { start_time, end_time } ISO 8601 bounds.
attribute_keys Optional root-span keys to include in each row, up to 20.
sort_by Optional { field, direction }; fields are created_at, span_name, duration, and llm_cost.
pagination { limit, cursorCreatedAt, cursorItemId }; limit is at most 50. Pass both cursor values from the previous result.

Filter objects use a field discriminator:

Field Operators and value
duration, llm_cost =, !=, <, <=, >, or >= with a number
error, span_name, customer_id, customer_user_id, session_id, dataset_id =, !=, contains, does_not_contain, beginsWith, endsWith, exists, or is_absent with a string; in with a string array
tags, rules_invoked any with a string array
behaviors any with behavior ID strings
span_attributes_roots A key plus a string operator and string value, or in with a string array
full_text_search contains with a string
calibration_unreviewed A judge_name

Use full_text_search first for content across span names, inputs, and outputs. It requires time_range.start_time and a window of at most 30 days. The default created_at desc sort works across history; every other sort requires a bounded window of at most seven days. Empty results do not prove the project has no traces—broaden the filter, inspect list_projects, or continue pagination.

Sessions

Tool Access Inputs Purpose
search_sessions Read filters, time_range, sort_by, pagination Search sessions by ID, trace count, latency, cost, or behavior IDs.
get_session_detail Read session_id Read timestamps, trace count, latency, cost, and token usage.
get_session_trace_ids Read session_id Read all trace IDs in a session.
get_session_behaviors Read session_id Read the latest session-scoped behavior result for each judge.
get_session_trace_behaviors Read session_id Read behavior results grouped across the session’s traces.

Views

Tool Access Inputs Purpose
list_views Read object_type List saved trace, session, span, or dashboard views.
get_view Read view_id Read one view and its configuration.
create_view Write name, description, object_type, visibility, config Create a personal or organization-visible view. Organization visibility requires the developer role.
update_view Write view_id, name, description, visibility, config Update an authored view; organization visibility requires the developer role.
delete_view Write view_id Delete an authored view.

Behaviors

Tool Access Inputs Purpose
get_behavior_detail Read behavior_id, start_date, end_date Read one behavior, its judge configuration, and optional date-bounded statistics.
update_behavior Write · developer behavior_id, description Update a behavior description.

Judges

Tool Access Inputs Purpose
list_judges Read version List judges and their selected configuration. Defaults to the latest version; select production for production configurations and enabled behavior outputs.
get_judge Read judge_id, version Read one selected judge version, including its definition, outputs, version metadata, and online-evaluation settings. Custom-judge source bundles are omitted.
list_judge_reasoning_levels Read — Organization-scoped list of enabled Agent Judge reasoning levels.
create_judge Write · developer judge Create a prompt or custom-code judge and its initial production version. Custom-code creation accepts a base64-encoded tar.gz bundle up to 1 MiB decoded. The judge type cannot change after creation.
update_judge Write · developer judge_id, update Update a judge description and/or create one complete immutable version. New versions specify the target version, existing judge and score types, complete implementation config, outputs, and monitoring-settings snapshot. Set promoteToProduction to promote the version.
delete_judges Write · developer judge_ids Delete one or more judges and their attached behaviors.

Prompts

Tool Access Inputs Purpose
list_prompts Read — List prompts with version counts and update times.
get_prompt Read prompt_name, commit_id, tag Read the latest prompt or select a version by commit or tag.
get_prompt_versions Read prompt_name List committed versions newest first.
commit_prompt Write · developer prompt_name, prompt, tags Create a prompt or commit a new version.
tag_prompt Write · developer prompt_name, commit_id, tags Add tags to a committed version.
untag_prompt Write · developer prompt_name, tags Remove tags without deleting versions.

Agents

Tool Access Inputs Purpose
list_agents Read — List custom agents and schedule metadata.
get_agent Read agent_config_id Read one custom-agent configuration.
create_agent Write · developer name, description, instructions, trigger, agent_mode, tool_permission_mode, slack Create a custom agent with optional schedule and Slack delivery.
update_agent Write · developer agent_config_id, name, description, instructions, trigger, agent_mode, tool_permission_mode, slack Update a custom-agent configuration.

Agent threads

Tool Access Inputs Purpose
list_agent_threads Read agent_type, agent_name, scope, limit List owner or project-visible threads for one agent.
get_agent_thread Read thread_id Read one thread transcript, metadata, and run status.
run_agent Write message, instructions_override, agent_type, agent_name, agent_config_id, thread_id Start or continue Judgment Agent or a custom agent and return thread/run IDs immediately.
set_agent_thread_project Write · developer thread_id, project_id Organization-scoped assignment of an unscoped thread to a project.

Datasets

Tool Access Inputs Purpose
list_datasets Read — List current datasets with versions and item counts.
get_dataset Read dataset_id, name, version, limit, cursor_created_at, cursor_example_id Read current dataset metadata and one keyset-paginated item page.
get_dataset_versions Read dataset_id List current dataset versions.
create_dataset Write · developer name, schema, examples Create a dataset, optionally with schema and initial examples.
add_examples_to_dataset Write · developer dataset_id, examples Add JSON examples or trace-backed examples and create a version.
remove_examples_from_dataset Write · developer dataset_id, example_ids Remove examples and create a version.
delete_dataset Write · admin dataset_id Delete an unreferenced dataset.
list_legacy_datasets Read — Deprecated compatibility list for legacy datasets.
get_legacy_dataset Read dataset_id, version, limit, cursor_sort_value, cursor_item_id Deprecated compatibility read for a legacy dataset page.
get_legacy_dataset_versions Read dataset_id Deprecated compatibility list of legacy versions.
get_legacy_dataset_item_ids Read dataset_id, version Deprecated compatibility list of legacy item IDs.

Tests

Tool Access Inputs Purpose
list_test_configs Read dataset_id List current test configurations, optionally for one dataset.
get_test_config Read test_config_id Read one dataset-and-judges test configuration.
create_test_config Write · developer name, description, dataset_id, judge_ids Create a reusable test configuration.
update_test_config Write · developer test_config_id, name, description Update a test configuration’s name or description.
delete_test_config Write · admin test_config_id Delete a test configuration.
list_test_runs Read test_config_id, dataset_id, status, limit List current test runs and pinned dataset versions.
get_test_run Read test_run_id Read one test run.
get_test_run_items Read test_run_id, limit, cursor Read per-example results, including partial results while work is running; page size is at most 200.
get_test_run_graph Read test_run_id Read aggregate judge graph data for one run.
run_test Write · developer test_config_id, dataset_version_number, judge_versions Queue a test from a saved configuration.
cancel_test_run Write · developer test_run_id Cancel a pending or running test.
delete_test_run Write · admin test_run_id Delete a test run.
list_legacy_tests Read limit, offset Deprecated compatibility list for legacy experiments.
get_legacy_test Read test_id Deprecated compatibility read for one legacy experiment.
get_legacy_test_items Read experiment_run_id, limit, cursor_sort_value, cursor_item_id Deprecated compatibility read for legacy per-example results.
get_legacy_test_graph Read experiment_run_id Deprecated compatibility read for legacy graph data.

Automations

Tool Access Inputs Purpose
list_automations Read — List conditions, actions, and active state.
get_automation Read rule_id Read one automation.
create_automation Write · developer name, description, conditions, combine_type, actions, cooldown_period, trigger_frequency Create an automation.
update_automation Write · developer rule_id, name, description, conditions, combine_type, actions, active, cooldown_period, trigger_frequency Update or enable/disable an automation.
delete_automation Write · admin rule_id Delete an automation.

Production issues

Tool Access Inputs Purpose
list_issues Read user_status, production_activity, priority List production issues with optional status, activity, and priority filters.
get_issue Read issue_id Read one issue, evidence, and event history.
write_issues Write · developer agent_thread_id, issues Create, update, or record recurrence for evidence-backed issues.
update_issue Write · developer issue_id, priority, user_status Update issue priority or user status.
resolve_issue Write · developer issue_id Resolve an issue and record the event.
ignore_issue Write · developer issue_id Ignore an issue and record the event.
reopen_issue Write · developer issue_id Reopen an issue and record the event.
write_issue_comment Write · developer issue_id, message Add a comment event without changing status.

Attribute mappers

Attribute mappers copy incoming span attributes to public Judgment attribute keys during ingestion.

Tool Access Inputs Purpose
list_mappers Read — List project and applicable organization-wide mappers.
get_mapper Read config_id Read one mapper.
create_mapper Write · developer name, rules, enabled, override, remove_source Create a project mapper. Organization-wide creation is admin-only.
update_mapper Write · developer config_id, name, rules, enabled, override, remove_source Update mapper rules or flags.
delete_mapper Write · developer config_id Delete a mapper.

Agent memory

Tool Access Inputs Purpose
list_agent_memory_entries Read — List project memory folders and files.
search_agent_memory_files Read query, limit Search memory paths and bodies for concise references.
fetch_agent_memory_files Read ids, names Fetch full files by ID or path, up to 200 total.
write_memory Write · developer name, body Create or update a durable project memory file.

Documentation bridge

These two read-only tools are registered on the authenticated data MCP for convenience, so coding agents can consult current product guidance without a second integration. Both proxy the docs site’s own hosted MCP server at docs.judgmentlabs.ai/mcp (its search_docs and get_page tools), so their inputs and results match that server exactly.

Tool Access Inputs Purpose
search_docs Read query, contentTypes, filters, limit Full-text search over Judgment documentation; returns each hit’s title, route, content type, and excerpt.
read_doc_page Read route Read one documentation page as agent-optimized Markdown.

Verify the live schema

If a field or tool differs from this page, inspect the tool list exposed to the current connection. Available tools can vary by authorization scope and account configuration. Reconnect after an authorization-scope change so the client refreshes tools/list.

Was this page helpful?