PromptScorer
Complete reference for the PromptScorer Python SDK
PromptScorer API Reference
PromptScorer is a special type of scorer where you can specify the prompt that the judge will use to make evaluations.
Authentication
Set up your credentials using environment variables:
export JUDGMENT_API_KEY="your_api_key_here"
export JUDGMENT_ORG_ID="your_organization_id_here"Class Methods
PromptScorer.create()
Initialize a PromptScorer object.
Parameters
name
str
Required
The name of the PromptScorer
prompt
str
Required
The prompt used by the LLM judge to make an evaluation
options
dict
Optional
If specified, the LLM judge will pick from one of the choices, and the score will be the one corresponding to the choice
judgment_api_key
str
Optional
Recommended - set using the JUDGMENT_API_KEY environment variable
organization_id
str
Optional
Recommended - set using the JUDGMENT_ORG_ID environment variable
Example Code
from judgeval.scorers import PromptScorer
scorer = PromptScorer.create(
name="Test Scorer",
prompt="Is the response positive or negative? Response: {{actual_output}}",
options={"positive" : 1, "negative" : 0}
)PromptScorer.get()
Retrieve a PromptScorer object that had already been created for the organization.
Parameters
name
str
Required
The name of the PromptScorer you would like to retrieve
judgment_api_key
str
Optional
Recommended - set using the JUDGMENT_API_KEY environment variable
organization_id
str
Optional
Recommended - set using the JUDGMENT_ORG_ID environment variable
Example Code
from judgeval.scorers import PromptScorer
scorer = PromptScorer.get(
name="Test Scorer"
)Instance Methods
scorer.append_to_prompt()
Add to the prompt for your PromptScorer
Parameters
prompt_addition
str
Required
This string will be added to the existing prompt for the scorer.
Example Code
scorer.append_to_prompt("Consider the overall tone, word choice, and emotional sentiment when making your determination.")scorer.set_threshold()
Update the threshold for your PromptScorer
Parameters
threshold
float
Required
The new threshold you would like the PromptScorer to use
Example Code
scorer.set_threshold(0.5)scorer.set_prompt()
Update the prompt for your PromptScorer
Parameters
prompt
str
Required
The new prompt you would like the PromptScorer to use
Example Code
scorer.set_prompt("Is the response helpful? Response: {{actual_output}}")scorer.set_options()
Update the options for your PromptScorer
Parameters
options
dict
Required
The new options you would like the PromptScorer to use
Example Code
scorer.set_options({"Yes" : 1, "No" : 0})scorer.get_threshold()
Retrieve the threshold for your PromptScorer
Parameters
None
Example Code
threshold = scorer.get_threshold()scorer.get_prompt()
Retrieve the prompt for your PromptScorer
Parameters
None
Example Code
prompt = scorer.get_prompt()scorer.get_options()
Retrieve the options for your PromptScorer
Parameters
None
Example Code
options = scorer.get_options()scorer.get_name()
Retrieve the name for your PromptScorer
Parameters
None
Example Code
name = scorer.get_name()scorer.get_config()
Retrieve the name, prompt, options, and threshold for your PromptScorer in a dictionary format
Parameters
None
Example Code
config = scorer.get_config()