Code judge bundle

How to write, package and upload a Python Code judge, or a new version of one.

A Code judge is Python you write and upload. Judgment installs its requirements and runs it on traces like any other judge. You create a Code judge, or a new version of one, by uploading a bundle through the API or the MCP create_judge and update_judge tools. The app can’t create Code judges.

The judge class

Subclass Judge from judgeval and type it with the response you return.

Response	`response_type`	`value`
`BinaryResponse`	`binary`	`bool`
`CategoricalResponse` subclass	`categorical`	`str`, one of the class’s `categories`
`NumericResponse`	`numeric`	`float`

Every response takes value and reason. A CategoricalResponse subclass must define a categories class variable listing every allowed Category(value=..., description=...). Import Category from judgeval.hosted.responses.

New Code judges must be categorical. BinaryResponse and NumericResponse are only for new versions of existing Code judges of that type.

from judgeval.data import Example
from judgeval.hosted.responses import Category
from judgeval.judges import CategoricalResponse, Judge


class EstimateResponse(CategoricalResponse):
    categories = [
        Category(value="Total stated", description="The recommendation states an order total."),
        Category(value="Total missing", description="The recommendation does not state an order total."),
    ]


class EstimatePresent(Judge[EstimateResponse]):
    async def score(self, data: Example) -> EstimateResponse:
        output = str(data["actual_output"])
        has_total = "$" in output and "total" in output.lower()
        return EstimateResponse(
            value="Total stated" if has_total else "Total missing",
            reason=(
                "The recommendation states an order total."
                if has_total
                else "The recommendation does not state an order total."
            ),
        )

score() receives an Example; read fields with bracket notation (data["input"], data["actual_output"]).

Bundle layout

A gzip-compressed tar file containing the entrypoint and, optionally, a requirements file.

tar -czf estimate-present.tar.gz estimate_present.py requirements.txt

Rule	Detail
Format	`tar.gz`
Size	At most 1 MiB
Entrypoint	`entrypoint_path` must match the file’s path inside the archive
Requirements	Optional. Judgment installs them before loading the class. Don’t list `judgeval`; the runtime provides it.
Credentials	The runtime doesn’t expose Judgment credentials to your code. Put other keys in Environment variables under Evaluation method on the judge’s Details tab.

Upload metadata

Field	Type	Required	Description
`scorer_name`	`string`	Yes	Name of the Code judge. A name that doesn’t exist in the project creates a new Code judge.
`entrypoint_path`	`string`	Yes	Path to the Python file inside the bundle.
`class_name`	`string`	Yes	The `Judge` subclass to load.
`response_type`	`binary \| categorical \| numeric`	Yes	Must match the class’s response. Can’t change after the judge exists.
`requirements_path`	`string`	No	Path to the requirements file inside the bundle.
`scorer_type`	`example \| trace`	No	Input the judge expects. Defaults to `example`.
`version`	`number`	No	`judgeval` SDK major version the code targets. Current default: `4`.
`categories`	`[{ value, description }]`	For `categorical`	At least two, matching the class’s `categories`.
`bump_major`	`boolean`	No	On re-upload, create the next major version instead of the next minor.

Upload with the API

POST /v1/projects/{project_id}/scorers/custom/bundle as multipart/form-data with two parts: metadata (JSON) and bundle (the file). Requires the developer role.

curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer ${JUDGMENT_API_KEY}" \
  --header "X-Organization-Id: ${JUDGMENT_ORG_ID}" \
  --form 'metadata={"scorer_name":"EstimatePresent","entrypoint_path":"estimate_present.py","requirements_path":"requirements.txt","class_name":"EstimatePresent","response_type":"categorical","categories":[{"value":"Total stated","description":"The recommendation states an order total."},{"value":"Total missing","description":"The recommendation does not state an order total."}],"version":4};type=application/json' \
  --form 'bundle=@estimate-present.tar.gz;type=application/gzip' \
  "https://api.judgmentlabs.ai/v1/projects/${JUDGMENT_PROJECT_ID}/scorers/custom/bundle"

JUDGMENT_PROJECT_ID is the project’s ID, shown in the Project ID column on Projects.

Response 200:

{
  "scorer_name": "EstimatePresent",
  "status": "success",
  "message": "Custom scorer 'EstimatePresent' uploaded successfully"
}

Status	Cause
`409`	A judge with this name exists and isn’t a Code judge.
`422`	Invalid metadata or bundle, a changed `response_type`, fewer than two categories for `categorical`, or a new name with a `response_type` other than `categorical` (“New judges must use categorical outputs”).
`404`	The project or judge wasn’t found.

The older POST /v1/projects/{project_id}/scorers/custom returns 410; use the bundle endpoint.

What an upload does

Uploading with a new scorer_name creates a categorical Code judge. Its first version is the production version, with online monitoring off.

Uploading with an existing scorer_name:

Creates the next minor version (or next major with bump_major: true).
Promotes it to production.
Keeps the latest version’s online monitoring settings: on or off, environments, sampling rate, run conditions and session scoring.

Upload with MCP

To create a Code judge, call create_judge with judgeType: "custom", a categorical initialVersion with at least two outputs, and the same config fields as below.

To upload a new version, call update_judge with a newVersion that has judgeType: "custom":

Field	Description
`judge_id`	The existing Code judge.
`newVersion.scoreType`	The judge’s existing type: `binary`, `numeric`, or `categorical` with the complete `outputs` list.
`newVersion.targetVersion`	The `major` and `minor` number of the new version.
`newVersion.promoteToProduction`	Whether the new version becomes the production version.
`newVersion.advancedSettings`	The complete online monitoring settings for the new version.
`newVersion.config.bundleBase64`	The `tar.gz` file, base64-encoded. Decoded size at most 1 MiB.
`newVersion.config.entrypointPath`	Path to the entrypoint inside the bundle.
`newVersion.config.className`	The `Judge` subclass.
`newVersion.config.requirementsPath`	Optional.
`newVersion.config.scorerType`	Optional. `example` (default) or `trace`.
`newVersion.config.sdkVersion`	Optional. Defaults to `4`.

See MCP tools.

After upload

Set online monitoring with newVersion.advancedSettings on update_judge, or under Run settings on the judge’s Details tab. The options are described in Run a judge on live traffic.
Add runtime keys under Environment variables in the Evaluation method section of the judge’s Details tab. Values are stored encrypted and supplied only while the judge runs.