Judgment Labs Logo
DocumentationSDK ReferenceChangelog
Judgeval Python-v1 SDK

CustomScorer

Reference server-hosted custom scorers created on the Judgment platform.

The CustomScorer class allows you to reference custom scorers that have been created and hosted on the Judgment platform. Custom scorers are server-hosted scoring functions that run on the platform, enabling you to use complex scoring logic without implementing it client-side.

from judgeval import Judgeval
from judgeval.v1.data.example import Example
from openai import OpenAI

client = Judgeval(project_name="default_project")
tracer = client.tracer.create()
openai = OpenAI()
openai = tracer.wrap(openai)

# Get a custom scorer by name
custom_scorer = client.scorers.custom_scorer.get(name="ResolutionScorer")

@tracer.observe(span_type="function")
def handle_customer_query(query: str) -> str:
    # Process the query
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}]
    )
    result = response.choices[0].message.content
    
    # Evaluate with custom scorer within the observed function
    tracer.async_evaluate(
        scorer=custom_scorer,
        example=Example.create(
            input=query,
            actual_output=result
        )
    )
    
    return result

result = handle_customer_query("Where is my package?")
print(result)

CustomScorerFactory.get()

Retrieve a custom scorer.

client.scorers.custom_scorer.get(
  name: str,
  class_name: Optional[str] = None,
)

Parameters

namerequired

:str

The name of the custom scorer to retrieve. This must match the name of a custom scorer that has been created on the Judgment platform.

class_name

:Optional[str]

The class name identifier for the scorer. If not provided, defaults to the name. This should match the class name used when the custom scorer was created on the platform.

Returns

A CustomScorer instance that can be used in evaluations.

Example

get_custom_scorer.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

# Get a custom scorer by name
scorer = client.scorers.custom_scorer.get(name="ResolutionScorer")

# Use with a specific class name if the class name is different from the name
scorer_with_class = client.scorers.custom_scorer.get(
    name="Resolution Scorer",
    class_name="ResolutionScorer"
)

Return Types

CustomScorer

Class instances returned by CustomScorerFactory.get() that represent server-hosted custom scorers.

The CustomScorer object contains the following properties and methods:

get_name()

:str

Returns the name of the custom scorer

get_class_name()

:str

Returns the class name identifier of the custom scorer

is_server_hosted()

:bool

Returns whether the scorer is server-hosted (always True for CustomScorer)

to_dict()

:Dict[str, Any]

Converts the scorer to a dictionary format for API serialization

Usage Examples

Using Custom Scorer with Tracer

tracer_custom_scorer.py
from judgeval import Judgeval
from judgeval.v1.data.example import Example
from openai import OpenAI

client = Judgeval(project_name="default_project")
tracer = client.tracer.create()
openai = OpenAI()
openai = tracer.wrap(openai)

# Get a custom scorer created on the platform
custom_scorer = client.scorers.custom_scorer.get(name="ResolutionScorer")

@tracer.observe(span_type="function")
def handle_customer_request(query: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}]
    )
    result = response.choices[0].message.content
    
    # Evaluate with custom scorer
    tracer.async_evaluate(
        scorer=custom_scorer,
        example=Example.create(
            input=query,
            actual_output=result
        )
    )
    
    return result

result = handle_customer_request("Where is my package?")
print(result)

Using Custom Scorer in Real-Time Evaluation

Custom scorers are primarily used with tracer.async_evaluate() for real-time evaluation during tracing. The scoring logic runs server-side on the Judgment platform.

realtime_custom_scorer.py
from judgeval import Judgeval
from judgeval.v1.data.example import Example
from openai import OpenAI

client = Judgeval(project_name="default_project")
tracer = client.tracer.create()
openai = OpenAI()
openai = tracer.wrap(openai)

# Get custom scorer
custom_scorer = client.scorers.custom_scorer.get(name="ResolutionScorer")

@tracer.observe(span_type="function")
def handle_customer_query(query: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}]
    )
    result = response.choices[0].message.content
    
    # Evaluate with custom scorer in real-time
    tracer.async_evaluate(
        scorer=custom_scorer,
        example=Example.create(
            input=query,
            actual_output=result
        )
    )
    
    return result

result = handle_customer_query("Where is my package?")
print(result)

Combining Custom Scorers with Built-in Scorers

You can use custom scorers alongside built-in scorers in real-time evaluations:

mixed_scorers.py
from judgeval import Judgeval
from judgeval.v1.data.example import Example
from openai import OpenAI

client = Judgeval(project_name="default_project")
tracer = client.tracer.create()
openai = OpenAI()
openai = tracer.wrap(openai)

# Get custom scorer
custom_scorer = client.scorers.custom_scorer.get(name="ResolutionScorer")

# Get built-in scorer
relevancy_scorer = client.scorers.built_in.answer_relevancy()

@tracer.observe(span_type="function")
def process_query(query: str) -> str:
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": query}]
    )
    result = response.choices[0].message.content
    
    example = Example.create(input=query, actual_output=result)
    
    # Evaluate with both scorers
    tracer.async_evaluate(scorer=custom_scorer, example=example)
    tracer.async_evaluate(scorer=relevancy_scorer, example=example)
    
    return result

result = process_query("Where is my package?")
print(result)

Next Steps

PromptScorer

Evaluate agent behavior based on a rubric you define and iterate on the platform.

Prompt

Create and iterate on prompts that you can dynamically load into your production environments.

Last updated on

On this page

CustomScorerFactory.get()ParametersReturnsExampleReturn TypesCustomScorerUsage ExamplesUsing Custom Scorer with TracerUsing Custom Scorer in Real-Time EvaluationCombining Custom Scorers with Built-in ScorersNext Steps