Code Judges

Implement the judge

Subclass Judge and choose the response type as its generic parameter. This example creates a binary judge that checks whether a support response includes the promised package date.

Install the example’s SDK and date-parsing dependency:

python -m pip install judgeval dateparser

import os

from dateparser.search import search_dates
from judgeval import Judgeval
from judgeval.data import Example
from judgeval.hosted.responses import BinaryResponse
from judgeval.judges import Judge


class IncludesPackageDate(Judge[BinaryResponse]):
    async def score(self, data: Example) -> BinaryResponse:
        output_field = os.environ.get("PACKAGE_DATE_FIELD", "actual_output")
        actual_output = str(data[output_field])
        includes_date = bool(search_dates(actual_output))

        return BinaryResponse(
            value=includes_date,
            reason=(
                "The response includes the promised package date."
                if includes_date
                else "The response does not include a package date."
            ),
        )


if __name__ == "__main__":
    client = Judgeval(project_name="my-project")

    results = client.evaluation.create().run(
        examples=[
            Example.create(
                input="When will my package arrive?",
                actual_output="Your package will arrive tomorrow at 10:00 AM.",
            )
        ],
        scorers=[IncludesPackageDate()],
        eval_run_name="package-date-code-judge-check",
    )

    scorer = results[0].scorers_data[0]
    print(
        {
            "name": scorer.name,
            "value": scorer.value,
            "score_type": scorer.score_type,
            "error": scorer.error,
        }
    )

score() receives an Example and returns one response with a value and reason. Use BinaryResponse for true or false, NumericResponse for a number, or a CategoricalResponse subclass with an explicit category list. The PACKAGE_DATE_FIELD setting also shows how the same code can read a judge-scoped environment variable when it runs in Judgment.

Run the evaluation

Run the file from the environment where your Judgment credentials are set:

python package-date-judge.py

Judgeval executes the Code Judge in this Python process and sends the evaluation run to the configured Judgment project. Install every imported dependency in the same local environment before running it.

Verify the result

Evaluation.run() returns one ScoringResult per example. Each result’s scorers_data list contains one ScorerData per judge. Inspect the first example’s first scorer as shown above and confirm that:

• name is IncludesPackageDate
• value is "Yes"
• score_type is "binary"
• error is None

Change actual_output to a response without a date and run the file again. The value should become "No" while error remains None. That contrast verifies the judge’s decision boundary rather than only proving that the code executed.

Upload the hosted judge

Local execution does not create a hosted judge asset. To let Judgment execute the class, package the entrypoint with a requirements file. Do not add judgeval to this file; the hosted runtime installs the SDK version declared in the upload metadata.

dateparser==1.2.2

Create a gzip-compressed tar bundle with both files at the archive root:

tar -czf package-date-code-judge.tar.gz \
  package-date-judge.py requirements.txt

Upload the bundle through the current multipart API. version: 4 selects the current Judgeval Code Judge contract, and the entrypoint and requirements paths must match their paths inside the archive.

curl --fail-with-body \
  --request POST \
  --header "Authorization: Bearer ${JUDGMENT_API_KEY}" \
  --header "X-Organization-Id: ${JUDGMENT_ORG_ID}" \
  --form 'metadata={"scorer_name":"IncludesPackageDate","entrypoint_path":"package-date-judge.py","requirements_path":"requirements.txt","class_name":"IncludesPackageDate","scorer_type":"example","response_type":"binary","version":4};type=application/json' \
  --form 'bundle=@package-date-code-judge.tar.gz;type=application/gzip' \
  "https://api.judgmentlabs.ai/v1/projects/${JUDGMENT_PROJECT_ID}/scorers/custom/bundle"

A successful response reports status: "success". The same supported upload is available to connected agents through the Judgment MCP create_judge tool. Select the custom judge type and pass the tar bundle as base64 in judge.config.bundleBase64. Create later immutable versions with update_judge. The deprecated Judgment CLI is not required for either path.

Configure hosted dependencies and environment variables

Judgment installs requirements.txt before loading the uploaded class. For configuration or third-party API keys that the class reads at runtime:

1. Open Judges > IncludesPackageDate.
2. Open Environment Variables.
3. Paste PACKAGE_DATE_FIELD=actual_output and press Enter.

Saved values are masked in the platform and supplied only while the hosted judge runs. Do not add JUDGMENT_API_KEY or JUDGMENT_ORG_ID here; the hosted runtime does not expose Judgment credentials to uploaded judge code.

Use the hosted judge online

Pass the uploaded name, rather than a local class instance, inside an observed function. This complete example initializes tracing, creates one agent span, and queues the server-side evaluation when that span completes:

from judgeval import Tracer


Tracer.init(project_name="my-project")


@Tracer.observe(span_type="agent", span_name="answer_support_question")
def answer_support_question(question: str) -> str:
    answer = "Your package will arrive tomorrow at 10:00 AM."
    Tracer.async_evaluate(
        "IncludesPackageDate",
        {"input": question, "actual_output": answer},
    )
    return answer


print(answer_support_question("When will my package arrive?"))
Tracer.shutdown()

Run the instrumented agent, open its completed trace in Logs > Traces, and inspect the IncludesPackageDate result. Judgment downloads the uploaded bundle, installs its declared dependencies, applies its saved environment variables, and executes the hosted class asynchronously.

For continuous online monitoring, link the binary output as a behavior and set the judge to Continuous with the intended sampling and span triggers. Follow Create and monitor a behavior to verify one result from new matching traffic.