--- title: "Code Judges" description: "Use Python to evaluate examples with deterministic logic, libraries, or custom model calls." sidebar: label: "Code judges" seo: title: "Code Judges | Judgment Docs" description: "Learn how Code Judges evaluate examples, then build, test, and upload one for hosted evaluation." --- A Code Judge evaluates an example with Python and returns a typed result. Use one when the criterion is best expressed with deterministic logic, a library, or custom model calls. Code Judges run locally during development or as hosted judge assets in Judgment. ## Create a Code Judge This workflow implements and verifies a Code Judge locally, then uploads it for hosted evaluation. ### Prerequisites - Python with the current `judgeval` package installed - `JUDGMENT_API_KEY`, `JUDGMENT_ORG_ID`, and the target project configured - The target `JUDGMENT_PROJECT_ID` when you upload the hosted judge - One representative input and actual output to evaluate 1. ### Implement the judge Subclass `Judge` and choose the response type as its generic parameter. This example creates a binary judge that checks whether a support response includes the promised package date. Install the example's SDK and date-parsing dependency: ```bash python -m pip install judgeval dateparser ``` ```python title="package-date-judge.py" import os from dateparser.search import search_dates from judgeval import Judgeval from judgeval.data import Example from judgeval.hosted.responses import BinaryResponse from judgeval.judges import Judge class IncludesPackageDate(Judge[BinaryResponse]): async def score(self, data: Example) -> BinaryResponse: output_field = os.environ.get("PACKAGE_DATE_FIELD", "actual_output") actual_output = str(data[output_field]) includes_date = bool(search_dates(actual_output)) return BinaryResponse( value=includes_date, reason=( "The response includes the promised package date." if includes_date else "The response does not include a package date." ), ) if __name__ == "__main__": client = Judgeval(project_name="my-project") results = client.evaluation.create().run( examples=[ Example.create( input="When will my package arrive?", actual_output="Your package will arrive tomorrow at 10:00 AM.", ) ], scorers=[IncludesPackageDate()], eval_run_name="package-date-code-judge-check", ) scorer = results[0].scorers_data[0] print( { "name": scorer.name, "value": scorer.value, "score_type": scorer.score_type, "error": scorer.error, } ) ``` `score()` receives an `Example` and returns one response with a value and reason. Use `BinaryResponse` for true or false, `NumericResponse` for a number, or a `CategoricalResponse` subclass with an explicit category list. The `PACKAGE_DATE_FIELD` setting also shows how the same code can read a judge-scoped environment variable when it runs in Judgment. 2. ### Run the evaluation Run the file from the environment where your Judgment credentials are set: ```bash python package-date-judge.py ``` Judgeval executes the Code Judge in this Python process and sends the evaluation run to the configured Judgment project. Install every imported dependency in the same local environment before running it. 3. ### Verify the result `Evaluation.run()` returns one `ScoringResult` per example. Each result's `scorers_data` list contains one `ScorerData` per judge. Inspect the first example's first scorer as shown above and confirm that: - `name` is `IncludesPackageDate` - `value` is `"Yes"` - `score_type` is `"binary"` - `error` is `None` Change `actual_output` to a response without a date and run the file again. The `value` should become `"No"` while `error` remains `None`. That contrast verifies the judge's decision boundary rather than only proving that the code executed. 4. ### Upload the hosted judge Local execution does not create a hosted judge asset. To let Judgment execute the class, package the entrypoint with a requirements file. Do not add `judgeval` to this file; the hosted runtime installs the SDK version declared in the upload metadata. ```text title="requirements.txt" dateparser==1.2.2 ``` Create a gzip-compressed tar bundle with both files at the archive root: ```bash tar -czf package-date-code-judge.tar.gz \ package-date-judge.py requirements.txt ``` Upload the bundle through the current multipart API. `version: 4` selects the current Judgeval Code Judge contract, and the entrypoint and requirements paths must match their paths inside the archive. ```bash curl --fail-with-body \ --request POST \ --header "Authorization: Bearer ${JUDGMENT_API_KEY}" \ --header "X-Organization-Id: ${JUDGMENT_ORG_ID}" \ --form 'metadata={"scorer_name":"IncludesPackageDate","entrypoint_path":"package-date-judge.py","requirements_path":"requirements.txt","class_name":"IncludesPackageDate","scorer_type":"example","response_type":"binary","version":4};type=application/json' \ --form 'bundle=@package-date-code-judge.tar.gz;type=application/gzip' \ "https://api.judgmentlabs.ai/v1/projects/${JUDGMENT_PROJECT_ID}/scorers/custom/bundle" ``` A successful response reports `status: "success"`. The same supported upload is available to connected agents through the Judgment MCP `create_judge` tool. Select the custom judge type and pass the tar bundle as base64 in `judge.config.bundleBase64`. Create later immutable versions with `update_judge`. The deprecated Judgment CLI is not required for either path. 5. ### Configure hosted dependencies and environment variables Judgment installs `requirements.txt` before loading the uploaded class. For configuration or third-party API keys that the class reads at runtime: 1. Open **Judges > IncludesPackageDate**. 2. Open **Environment Variables**. 3. Paste `PACKAGE_DATE_FIELD=actual_output` and press Enter. Saved values are masked in the platform and supplied only while the hosted judge runs. Do not add `JUDGMENT_API_KEY` or `JUDGMENT_ORG_ID` here; the hosted runtime does not expose Judgment credentials to uploaded judge code. 6. ### Use the hosted judge online Pass the uploaded name, rather than a local class instance, inside an observed function. This complete example initializes tracing, creates one agent span, and queues the server-side evaluation when that span completes: ```python from judgeval import Tracer Tracer.init(project_name="my-project") @Tracer.observe(span_type="agent", span_name="answer_support_question") def answer_support_question(question: str) -> str: answer = "Your package will arrive tomorrow at 10:00 AM." Tracer.async_evaluate( "IncludesPackageDate", {"input": question, "actual_output": answer}, ) return answer print(answer_support_question("When will my package arrive?")) Tracer.shutdown() ``` Run the instrumented agent, open its completed trace in **Logs > Traces**, and inspect the `IncludesPackageDate` result. Judgment downloads the uploaded bundle, installs its declared dependencies, applies its saved environment variables, and executes the hosted class asynchronously. For continuous online monitoring, link the binary output as a behavior and set the judge to **Continuous** with the intended sampling and span triggers. Follow [Create and monitor a behavior](/documentation/judges/behavior-setup) to verify one result from new matching traffic. > **Local and hosted judges run separately** > > Do not mix a local Code Judge instance and a hosted judge name in the same > `evaluation.create().run()` call. Use separate evaluation runs when you need > both. ## Choose the output deliberately - A binary Code Judge returns a true-or-false result. - A categorical Code Judge returns one configured category per result. - A numeric Code Judge returns a score within its defined range. See the [evaluation data model](/documentation/judges) for the complete output model. For exact response classes and citations, use the [Python Judge SDK reference](/sdk-reference/python/judges). ## Next step Add more representative examples and repeat the local evaluation before each upload. When the hosted binary result is stable, [create and monitor its behavior](/documentation/judges/behavior-setup), or add the judge by name to an [offline test](/documentation/tests/offline-tests).