Judgment Labs Logo
DocumentationSDK ReferenceChangelog
Judgeval Python-v1 SDK

Prompt

Create and iterate on prompts that you can dynamically load into your production environments.

The Prompt class provides an interface to easily create and version prompts that are stored on the Judgment platform. You can then dynamically retrieve any version of any prompt to be used in your application code.

Prompt's versioning system allows you to iterate on commits by creating multiple commits. You can also label prompt commits with tags that serve as a keywords to retrieve a specific version of a prompt.

from openai import OpenAI
from judgeval import Judgeval

# Initialize the Judgeval client
client = Judgeval(project_name="default_project")

# Create a prompt. If the prompt name already exists, a new commit will be created
client.prompts.create(
  name="Movie Recommendation Prompt",
  prompt="""Given this genre: {{genre}} and this decade: {{decade}},
  can you recommend a movie of that genre that was released in that decade?""",
  tags=["production"],
)

class MovieRecommendationAgent:
    def __init__(self):
        self.client = OpenAI()
        self.judgeval = Judgeval(project_name="default_project")

    def get_recommendation(self, user_genre: str, user_decade: str):
        prompt = self.judgeval.prompts.get(
          name="Movie Recommendation Prompt",
          tag="production"
        )
        if prompt is None:
            raise ValueError("Prompt not found")
        response = self.client.chat.completions.create(
            model="gpt-5.2",
            messages=[
              {
                "role": "user",
                "content": prompt.compile(genre=user_genre, decade=user_decade)
              }
            ]
        )
        return f"Response: {response.choices[0].message.content}"

if __name__ == "__main__":
    agent = MovieRecommendationAgent()
    agent.get_recommendation("horror", "90s")

PromptFactory.create()

Create a new prompt or create a new commit for an existing prompt.

client.prompts.create(
  name: str,
  prompt: str,
  tags: Optional[List[str]] = None,
)

Parameters

Each tag is unique to one commit per prompt. Reusing a tag moves it from its previous commit to the new one.

namerequired

:str

The name of the prompt, which is unique to each prompt. Specifying a name that already exists will create a new commit for that prompt

promptrequired

:str

The prompt contents to save. Use {{ mustache notation }} for variables to specify parts of the prompt that can be dynamically replaced at runtime using Prompt.compile()

tags

:Optional[List[str]]

If specified, the tags in this list will be attached to the created commit. Please read the warning in the Overview section for more details on tagging behavior

Returns

A Prompt instance representing the created prompt commit, or None if the project was not found during client initialization.

Example

create_prompt.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

prompt = client.prompts.create(
    name="Test Prompt",
    prompt="Recommend a movie in the {{genre}} genre",
    tags=["production"],
)

PromptFactory.get()

Retrieve a specific version of a Prompt that has already been created for the project.

  • Passing in a commit_id will retrieve the commit with that id for the prompt, if it exists.
  • Passing in a tag will retrieve the commit for the prompt with that tag attached, if it exists.
client.prompts.get(
  *,
  name: str,
  commit_id: Optional[str] = None,
  tag: Optional[str] = None,
)

Parameters

namerequired

:str

The name of the Prompt you would like to retrieve. Returns None if the prompt does not exist

commit_id

:Optional[str]

The ID of the prompt commit you would like to retrieve. Mutually exclusive with tag. Returns None if the prompt commit does not exist

tag

:Optional[str]

The tag associated with the prompt commit you would like to retrieve. Mutually exclusive with commit_id. Returns None if the prompt commit does not exist

Returns

A Prompt instance if found, or None if the prompt does not exist. The Prompt instance will have the following metadata fields:

  • creator_first_name: The first name of the user who created the prompt
  • creator_last_name: The last name of the user who created the prompt
  • creator_email: The email of the user who created the prompt

Example

get_prompt.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

prompt = client.prompts.get(name="Test Prompt")
if prompt is None:
    print("Prompt not found")

PromptFactory.tag()

Attach one or more tags to a specific version of a Prompt that has already been created for the project.

client.prompts.tag(
  name: str,
  commit_id: str,
  tags: List[str],
)

Parameters

Each tag is unique to one commit per prompt. Reusing a tag moves it from its previous commit to the new one.

namerequired

:str

The name of the Prompt you would like to tag. Throws a JudgmentException if the prompt does not exist

commit_idrequired

:str

The ID of the prompt commit you would like to attach tag(s) to. Throws a JudgmentException if the prompt commit does not exist

tagsrequired

:List[str]

The tag(s) you would like to add to the prompt commit. Throws a JudgmentException if this list is empty

Returns

A str representing the commit ID which the tag(s) were attached to, or None if the project was not found during client initialization.

Example

tag_prompt.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

commit_id = client.prompts.tag(
    name="Test Prompt",
    commit_id="some_commit_id",
    tags=["production", "v1.2"],
)

PromptFactory.untag()

Remove one or more tags attached to any of the versions of a Prompt that has already been created for the project.

client.prompts.untag(
  name: str,
  tags: List[str],
)

Parameters

namerequired

:str

The name of the Prompt you would like to untag. Throws a JudgmentException if the prompt does not exist

tagsrequired

:List[str]

The tag(s) you would like to remove from the prompt. Throws a JudgmentException if this list is empty or if the prompt does not have any of the provided tags

Returns

A List[str] representing the commit IDs which the tag(s) were removed from, or None if the project was not found during client initialization.

Example

untag_prompt.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

commit_ids = client.prompts.untag(
    name="Test Prompt",
    tags=["production"],
)

PromptFactory.list()

Retrieve all the versions of a Prompt that has already been created for the project.

client.prompts.list(
  name: str,
)

Parameters

namerequired

:str

The name of the Prompt whose versions you would like to retrieve. If the prompt does not exist, an empty list will be returned

Returns

A List[Prompt] representing the versions of the prompt, or None if the project was not found during client initialization. Each Prompt in the list will have the following metadata fields:

  • creator_first_name: The first name of the user who created the prompt
  • creator_last_name: The last name of the user who created the prompt
  • creator_email: The email of the user who created the prompt

Example

list_prompts.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

prompts = client.prompts.list(name="Test Prompt")

Return Types

Prompt

Class instances returned by PromptFactory.create(), PromptFactory.get(), and PromptFactory.list(), that provide details about a specific prompt version and additional methods for prompt management.

The Prompt object contains the following properties:

name

:str

Name of the prompt, which is unique between prompts

prompt

:str
Contents of the prompt

created_at

:str

Timestamp of when the prompt commit was created

tags

:List[str]
Tags attached to the prompt commit

commit_id

:str
ID of the retrieved or created commit

parent_commit_id

:Union[str, None]
Parent of the current commit

metadata

:Dict[str, str]

Metadata about the prompt. This could include the first name and last name of the user who created the prompt. For example:

{
  "creator_first_name": "John",
  "creator_last_name": "Doe",
  "creator_email": "john.doe@example.com",
}

Prompt.compile()

Compile a prompt template by substituting variables with provided values.

prompt.compile(**kwargs) -> str

Parameters

**kwargs

:Any

Keyword arguments where keys match variable names in the prompt template (using {{variable_name}} syntax). All variables in the template must be provided, otherwise a ValueError will be raised.

Returns

A str representing the compiled prompt with all variables substituted.

Throws

  • ValueError if a required variable is missing from the provided keyword arguments

Example

compile_prompt.py
from judgeval import Judgeval

client = Judgeval(project_name="default_project")

prompt = client.prompts.get(name="Test Prompt")
if prompt:
    compiled = prompt.compile(genre="horror", decade="90s")
    print(compiled)

CustomScorer

Reference server-hosted custom scorers created on the Judgment platform.

Example

A single evaluation example containing input data and expected outputs

Last updated on

On this page

PromptFactory.create()ParametersReturnsExamplePromptFactory.get()ParametersReturnsExamplePromptFactory.tag()ParametersReturnsExamplePromptFactory.untag()ParametersReturnsExamplePromptFactory.list()ParametersReturnsExampleReturn TypesPromptPrompt.compile()ParametersReturnsThrowsExample