---
title: "Code judge bundle"
sidebar:
  label: "Code judge bundle"
description: "How to write, package and upload a Python Code judge, or a new version of one."
seo:
  title: "Code judge bundle | Judgment Docs"
  description: "How to write, package and upload a Python Code judge, or a new version of one."
---

A Code judge is Python you write and upload. Judgment installs its requirements and runs it on traces like any other judge. You create a Code judge, or a new version of one, by uploading a bundle through the API or the MCP `create_judge` and `update_judge` tools. The app can't create Code judges.

## The judge class

Subclass `Judge` from `judgeval` and type it with the response you return.

| Response | `response_type` | `value` |
|---|---|---|
| `BinaryResponse` | `binary` | `bool` |
| `CategoricalResponse` subclass | `categorical` | `str`, one of the class's `categories` |
| `NumericResponse` | `numeric` | `float` |

Every response takes `value` and `reason`. A `CategoricalResponse` subclass must define a `categories` class variable listing every allowed `Category(value=..., description=...)`. Import `Category` from `judgeval.hosted.responses`.

New Code judges must be categorical. `BinaryResponse` and `NumericResponse` are only for new versions of existing Code judges of that type.

```python title="estimate_present.py"
from judgeval.data import Example
from judgeval.hosted.responses import Category
from judgeval.judges import CategoricalResponse, Judge


class EstimateResponse(CategoricalResponse):
    categories = [
        Category(value="Total stated", description="The recommendation states an order total."),
        Category(value="Total missing", description="The recommendation does not state an order total."),
    ]


class EstimatePresent(Judge[EstimateResponse]):
    async def score(self, data: Example) -> EstimateResponse:
        output = str(data["actual_output"])
        has_total = "$" in output and "total" in output.lower()
        return EstimateResponse(
            value="Total stated" if has_total else "Total missing",
            reason=(
                "The recommendation states an order total."
                if has_total
                else "The recommendation does not state an order total."
            ),
        )
```

`score()` receives an `Example`; read fields with bracket notation (`data["input"]`, `data["actual_output"]`).

## Bundle layout

A gzip-compressed tar file containing the entrypoint and, optionally, a requirements file.

```bash
tar -czf estimate-present.tar.gz estimate_present.py requirements.txt
```

| Rule | Detail |
|---|---|
| Format | `tar.gz` |
| Size | At most 1 MiB |
| Entrypoint | `entrypoint_path` must match the file's path inside the archive |
| Requirements | Optional. Judgment installs them before loading the class. Don't list `judgeval`; the runtime provides it. |
| Credentials | The runtime doesn't expose Judgment credentials to your code. Put other keys in **Environment variables** under **Evaluation method** on the judge's **Details** tab. |

## Upload metadata

| Field | Type | Required | Description |
|---|---|---|---|
| `scorer_name` | `string` | Yes | Name of the Code judge. A name that doesn't exist in the project creates a new Code judge. |
| `entrypoint_path` | `string` | Yes | Path to the Python file inside the bundle. |
| `class_name` | `string` | Yes | The `Judge` subclass to load. |
| `response_type` | `binary \| categorical \| numeric` | Yes | Must match the class's response. Can't change after the judge exists. |
| `requirements_path` | `string` | No | Path to the requirements file inside the bundle. |
| `scorer_type` | `example \| trace` | No | Input the judge expects. Defaults to `example`. |
| `version` | `number` | No | `judgeval` SDK major version the code targets. Current default: `4`. |
| `categories` | `[{ value, description }]` | For `categorical` | At least two, matching the class's `categories`. |
| `bump_major` | `boolean` | No | On re-upload, create the next major version instead of the next minor. |

## Upload with the API

`POST /v1/projects/{project_id}/scorers/custom/bundle` as `multipart/form-data` with two parts: `metadata` (JSON) and `bundle` (the file). Requires the developer role.

```bash
curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer ${JUDGMENT_API_KEY}" \
  --header "X-Organization-Id: ${JUDGMENT_ORG_ID}" \
  --form 'metadata={"scorer_name":"EstimatePresent","entrypoint_path":"estimate_present.py","requirements_path":"requirements.txt","class_name":"EstimatePresent","response_type":"categorical","categories":[{"value":"Total stated","description":"The recommendation states an order total."},{"value":"Total missing","description":"The recommendation does not state an order total."}],"version":4};type=application/json' \
  --form 'bundle=@estimate-present.tar.gz;type=application/gzip' \
  "https://api.judgmentlabs.ai/v1/projects/${JUDGMENT_PROJECT_ID}/scorers/custom/bundle"
```

`JUDGMENT_PROJECT_ID` is the project's ID, shown in the **Project ID** column on **Projects**.

Response `200`:

```json
{
  "scorer_name": "EstimatePresent",
  "status": "success",
  "message": "Custom scorer 'EstimatePresent' uploaded successfully"
}
```

| Status | Cause |
|---|---|
| `409` | A judge with this name exists and isn't a Code judge. |
| `422` | Invalid metadata or bundle, a changed `response_type`, fewer than two categories for `categorical`, or a new name with a `response_type` other than `categorical` ("New judges must use categorical outputs"). |
| `404` | The project or judge wasn't found. |

The older `POST /v1/projects/{project_id}/scorers/custom` returns `410`; use the bundle endpoint.

## What an upload does

Uploading with a new `scorer_name` creates a categorical Code judge. Its first version is the production version, with online monitoring off.

Uploading with an existing `scorer_name`:

- Creates the next minor version (or next major with `bump_major: true`).
- Promotes it to production.
- Keeps the latest version's online monitoring settings: on or off, environments, sampling rate, run conditions and session scoring.

## Upload with MCP

To create a Code judge, call `create_judge` with `judgeType: "custom"`, a categorical `initialVersion` with at least two `outputs`, and the same `config` fields as below.

To upload a new version, call `update_judge` with a `newVersion` that has `judgeType: "custom"`:

| Field | Description |
|---|---|
| `judge_id` | The existing Code judge. |
| `newVersion.scoreType` | The judge's existing type: `binary`, `numeric`, or `categorical` with the complete `outputs` list. |
| `newVersion.targetVersion` | The `major` and `minor` number of the new version. |
| `newVersion.promoteToProduction` | Whether the new version becomes the production version. |
| `newVersion.advancedSettings` | The complete online monitoring settings for the new version. |
| `newVersion.config.bundleBase64` | The `tar.gz` file, base64-encoded. Decoded size at most 1 MiB. |
| `newVersion.config.entrypointPath` | Path to the entrypoint inside the bundle. |
| `newVersion.config.className` | The `Judge` subclass. |
| `newVersion.config.requirementsPath` | Optional. |
| `newVersion.config.scorerType` | Optional. `example` (default) or `trace`. |
| `newVersion.config.sdkVersion` | Optional. Defaults to `4`. |

See [MCP tools](/documentation/reference/mcp-tools).

## After upload

- Set online monitoring with `newVersion.advancedSettings` on `update_judge`, or under **Run settings** on the judge's **Details** tab. The options are described in [Run a judge on live traffic](/documentation/judges/run-on-live-traffic).
- Add runtime keys under **Environment variables** in the **Evaluation method** section of the judge's **Details** tab. Values are stored encrypted and supplied only while the judge runs.