Code Judges

Create your judge file

You can use the CLI to generate skeleton code for your judge:

judgeval scorer init --response-type binary --name ResolutionScorer

This creates a resolution_scorer.py file with the appropriate boilerplate. Use --response-type to select binary, categorical, or numeric.

CLI Options

If you prefer to write it manually, here's an example:

from judgeval.v1.judges import Judge, BinaryResponse
from judgeval.v1.data import Example

class ResolutionScorer(Judge[BinaryResponse]):
    async def score(self, data: Example) -> BinaryResponse:
        actual_output = data.get_property("actual_output")

        if "package" in actual_output:
            return BinaryResponse(
                value=True,
                reason="The response contains package information."
            )

        return BinaryResponse(
            value=False,
            reason="The response does not contain package information."
        )

The score method:

• Takes an Example as input
• Can access trace data via data.trace and data.trace.spans when available
• Returns one of three response types:
  • BinaryResponse: For true/false evaluations with a value (bool) and reason (string)
  • CategoricalResponse: For categorical evaluations with a value (string) and reason (string)
  • NumericResponse: For numeric evaluations with a value (float) and reason (string)
• All response types support optional citations to reference specific spans in your trace

Here's an example that uses trace data to score based on span information:

from judgeval.v1.judges import Judge, NumericResponse
from judgeval.v1.data import Example

class ToolCallScorer(Judge[NumericResponse]):
    async def score(self, data: Example) -> NumericResponse:
        if not data.trace or not data.trace.spans:
            return NumericResponse(
                value=0.0,
                reason="No trace data available."
            )

        tool_calls = [span for span in data.trace.spans if span.get("span_kind") == "tool"]

        return NumericResponse(
            value=float(len(tool_calls)),
            reason=f"Agent made {len(tool_calls)} tool call(s)."
        )

Create a requirements file

Create a requirements.txt file with any dependencies your scorer needs. If you used judgeval scorer init with the --include-requirements flag to generate your judge, this file will already be created for you.

# Add any dependencies your scorer needs
# openai>=1.0.0
# numpy>=1.24.0

Install the CLI

The CLI is included with the judgeval package:

pip install judgeval

Set your credentials

Set your Judgment API key and organization ID as environment variables:

export JUDGMENT_API_KEY="your-api-key"
export JUDGMENT_ORG_ID="your-org-id"

Upload the judge

The entrypoint file is the file that contains the scorer class definition. It should be passed as an argument to the judgeval scorer upload command:

judgeval scorer upload my_scorer.py -p my_project

You can also include additional files or directories needed by your judge:

judgeval scorer upload my_scorer.py -i utils/ -i shared_helpers.py -p my_project

To provide a requirements file and a custom name:

judgeval scorer upload my_scorer.py -r requirements.txt -n "Resolution Scorer" -p my_project

To bump the major version of an existing judge:

judgeval scorer upload my_scorer.py -m -p my_project

Navigate to the Judges page within your project.

Click on the Code Judge you would like to add the environmental variables for and click the "Environment variables" button on the top right of the page

Enter in the environmental variable and click the "Add" button. Do this for all environmental variables needed for the Code Judge.