Judgeval

The main entry point for interacting with the Judgment platform.

Judgeval connects to your Judgment project and gives you access to SQL queries, evaluations, and datasets.

Credentials are resolved in order: explicit arguments first, then environment variables JUDGMENT_API_KEY, JUDGMENT_ORG_ID, and JUDGMENT_API_URL.

Minimal setup (credentials from environment variables):

from judgeval import Judgeval

client = Judgeval(project_name="search-assistant")

Explicit credentials:

client = Judgeval(
    project_name="search-assistant",
    api_key="jdg_...",
    organization_id="org_...",
)

Once initialized, use the evaluation and datasets properties:

eval_runner = client.evaluation.create()
dataset = client.datasets.get(name="golden-set")

Attributes

PropType

api?JudgmentApiClient

The /v1 API, one method per Judgment MCP tool.

TypeJudgmentApiClient

evaluation?Any

Access evaluations for scoring examples with hosted scorers or Code judges. Use .create() to get an Evaluation you can call .run() on.

TypeAny

test_suites?Any

Create test suites and edit their versioned cases. create and list test suites, add or edit cases in a suite's draft, and publish versions.

TypeAny

test_runs?Any

Run test suites and read their results. start a run, then read it, its results, or score it again.

TypeAny

offline_tests?Any

Deprecated. Use test_suites and test_runs instead. Access offline tests: test configs and dataset-backed test runs. Use .create_config() to bind a dataset to judges, and .run() to execute a test run (optionally driving an agent entrypoint and asserting a pass condition).

TypeAny

datasets?Any

Manage datasets of evaluation examples. Use .create(), .get(), or .list() to work with datasets.

TypeAny

prompts?Any

Manage versioned prompt templates with tagging support. Use .create(), .get(), .tag(), or .list() to work with prompts.

TypeAny

agent_judges?Any

Manage Agent judges (prompt-based scorers) on the platform. Use .create() or .update() to create and update prompt-based Agent judges.

TypeAny

discover_schema()

Return the SQL schema reference as Markdown.

Mirrors MCP discover_schema: published tables, column types and descriptions, row semantics, examples, and query limits. Fetches the server’s generated catalog using this client’s credentials. Contains no project data and does not require a resolved project.

print(client.discover_schema())

def discover_schema() -> str:

Returns

str - The virtual schema reference as a Markdown string.

sql()

Run one read-only SQL SELECT for this organization and project.

The server derives tenant scope from the client’s credentials and resolved project. Call discover_schema() for supported tables and columns. Requires viewer access.

Results are capped by the server at 1,000 rows and 5 MiB; exceeding either cap returns an error. Use SQL predicates and LIMIT to narrow results.

result = client.sql("SELECT count() AS run_count FROM telemetry.traces")
print(result["rows"])

def sql(sql_text) -> SqlResponse:

Parameters

PropType

sql_textstr

One SELECT against the virtual schema, at most 50,000 characters. Use SQL predicates to narrow the results.

Typestr

Returns

SqlResponse - A dictionary with catalog_version, columns (name, type, nullable), rows (dictionaries keyed by column name), row_count, and elapsed_ms. Integers outside JavaScript’s safe range arrive as exact decimal strings.