Prompt

Overview

The Prompt class provides an interface to easily create and version prompts that are stored on the Judgment platform. You can then dynamically retrieve any version of any prompt to be used in your application code.

Prompt's versioning system allows you to iterate on commits by creating multiple commits. You can also label prompt commits with tags that serve as a keywords to retrieve a specific version of a prompt.

Keep in mind that each tag is unique to one commit per prompt in your project. This leads to the following behavior:

When you call Prompt.create() and specify a tag that is already attached to a different commit for the same prompt, the tag will be moved from the original commit to your new commit.
When you call Prompt.tag() and specify a tag that is already attached to a different commit for the same prompt, the tag will be moved from the original commit to the commit specified in Prompt.tag()

Quick Start Example

create_and_use_prompt.py

from openai import OpenAI
from judgeval.prompt import Prompt

# Create a prompt. If the prompt name already exists, a new commit will be created
Prompt.create(
  project_name="default_project",
  name="Movie Recommendation Prompt",
  prompt="""Given this genre: {{genre}} and this decade: {{decade}},
  can you recommend a movie of that genre that was released in that decade?""",
  tags=["production"],
)

class MovieRecommendationAgent:
    def __init__(self):
        self.client = OpenAI()

    def get_recommendation(self, user_genre: str, user_decade: str):
        prompt = Prompt.get(
          project_name="default_project",
          name="Movie Recommendation Prompt",
          tag="production"
        )
        response = self.client.chat.completions.create(
            model="gpt-5", 
            messages=[
              {
                "role": "user",
                "content": prompt.compile(genre=user_genre, decade=user_decade)
              }
            ]
        )
        return f"Response: {response.choices[0].message.content}"

if __name__ == "__main__":
    agent = MovieRecommendationAgent()
    agent.get_recommendation("horror", "90s")

Authentication

Set up your credentials using environment variables:

export JUDGMENT_API_KEY="your_key_here"
export JUDGMENT_ORG_ID="your_org_id_here"

# Add to your .env file
JUDGMENT_API_KEY="your_key_here"
JUDGMENT_ORG_ID="your_org_id_here"

Prompt Creation & Retrieval

`Prompt.create()`

Initialize a Prompt object.

Parameters

`project_name`

str

Required

The name of the project to create a prompt in

`name`

str

Required

The name of the prompt, which is unique to each prompt. Specifying a name that already exists will create a new commit for that prompt

`prompt`

str

Required

The prompt contents to save. Use mustache notation ({{variable}}) to specify parts of the prompt that can be dynamically replaced at runtime using Prompt.compile()

`tags`

Optional[List[str]]

Optional

If specified, the tags in this list will be attached to the created commit. Please read the warning in the Overview section for more details on tagging behavior

`judgment_api_key`

str

Optional

Recommended - set using the JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

Prompt

Optional

A Prompt instance

Example Code

create_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.create(
    project_name="default_project",
    name="Test Prompt", 
    prompt="Recommend a movie in the {{genre}} genre",
    tags=["production"]
)

`Prompt.get()`

Retrieve the latest version of a Prompt that has already been created for the project. Passing in a commit_id will retrieve the commit with that id for the prompt, if it exists. Passing in a tag will retrieve the commit for the prompt with that tag attached, if it exists.

Parameters

`project_name`

str

Required

The name of the project to fetch the prompt from

`name`

str

Required

The name of the Prompt you would like to retrieve. Throws a JudgmentException if the prompt does not exist

`commit_id`

str

Optional

The ID of the prompt commit you would like to retrieve. Mutually exclusive with tag. Throws a JudgmentException if the prompt commit does not exist

`tag`

str

Optional

The tag associated with the prompt commit you would like to retrieve. Mutually exclusive with commit_id. Throws a JudgmentException if the prompt commit does not exist.

`judgment_api_key`

str

Optional

Recommended - set using the JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

Prompt

Optional

A Prompt instance The Prompt instance will have the following metadata fields:

creator_first_name: The first name of the user who created the prompt
creator_last_name: The last name of the user who created the prompt
creator_email: The email of the user who created the prompt

Example Code

get_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.get(
    project_name="default_project",
    name="Test Prompt"
)

`Prompt.tag()`

Attach one or more tags to a specific version of a Prompt that has already been created for the project. Please read the warning in the Overview section for more details on tagging behavior.

Parameters

`project_name`

str

Required

The name of the project that the prompt you would like to tag is in

`name`

str

Required

The name of the Prompt you would like to tag. Throws a JudgmentException if the prompt does not exist

`commit_id`

str

Optional

The ID of the prompt commit you would like to attach tag(s) to. Throws a JudgmentException if the prompt commit does not exist

`tags`

List[str]

Optional

The tag(s) you would like to add to the prompt commit. Throws a JudgmentException if this list is empty

`judgment_api_key`

str

Optional

Recommended - set using the JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

str

Optional

The commit ID which the tag(s) were attached to

Example Code

tag_prompt.py

from judgeval.prompt import Prompt

commit_id = Prompt.tag(
    project_name="default_project",
    name="Test Prompt",
    commit_id="some_commit_id",
    tags=["production", "v1.2"],
)

`Prompt.untag()`

Remove one or more tags attached to any of the versions of a Prompt that has already been created for the project.

Parameters

`project_name`

str

Required

The name of the project that the prompt you would like to untag is in

`name`

str

Required

The name of the Prompt you would like to untag. Throws a JudgmentException if the prompt does not exist

`tags`

List[str]

Optional

The tag(s) you would like to remove from the prompt. Throws a JudgmentException if this list is empty or if the prompt does not have any of the provided tags

`judgment_api_key`

str

Optional

Recommended - set using the JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

List[str]

Optional

The commit IDs which the tag(s) were removed from

Example Code

untag_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.untag(
    project_name="default_project",
    name="Test Prompt",
    tags=["production"],
)

`Prompt.list()`

Retrieve all the versions of a Prompt that has already been created for the project.

Parameters

`project_name`

str

Required

The name of the project to fetch the prompt from

`name`

str

Required

The name of the Prompt whose versions you would like to retrieve. If the prompt does not exist, an empty list will be returned.

`judgment_api_key`

str

Optional

Recommended - set using the JUDGMENT_API_KEY environment variable

`organization_id`

str

Optional

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

List[Prompt]

Optional

A list of Prompt instances Each Prompt instance will have the following metadata fields:

creator_first_name: The first name of the user who created the prompt
creator_last_name: The last name of the user who created the prompt
creator_email: The email of the user who created the prompt

Example Code

list_prompts.py

from judgeval.prompt import Prompt

prompts = Prompt.list(
    project_name="default_project",
    name="Test Prompt"
)

`Prompt`

Class instances returned by Prompt.create(), Prompt.get(), and Prompt.list(), that provide details about a specific prompt version and additional methods for prompt management.

Properties

`name`

str

Required

Name of the prompt, which is unique between prompts

`prompt`

str

Required

Contents of the prompt

`created_at`

str

Required

Timestamp of when the prompt commit was created

`tags`

List[str]

Required

Tags attached to the prompt commit

`commit_id`

str

Required

ID of the retrieved or created commit

`parent_commit_id`

Union[str, None]

Required

Parent of the current commit

`metadata`

Dict[str, str]

Required

Metadata about the prompt. This could include the first name and last name of the user who created the prompt. For example:

{
  "creator_first_name": "John",
  "creator_last_name": "Doe",
  "creator_email": "john.doe@example.com",
}

Usage Pattern

# `Prompt.create()` and `Prompt.get()` return Prompt instances
prompt_instance = Prompt.create(
    name="movie_recommendation_prompt", 
    prompt="Recommend a movie in the {{genre}} genre",
)

retrieved_prompt = Prompt.get(name="movie_recommendation_prompt")

# Both return Prompt instances with the compile method
print(prompt_instance.compile(genre="horror"))  # Prints "Recommend a movie in the horror genre"
print(retrieved_prompt.compile(genre="horror"))  # Prints "Recommend a movie in the horror genre"

Prompt

Parameters

Returns

Example Code

Parameters

Returns

Example Code

Parameters

Returns

Example Code

Parameters

Returns

Example Code

Parameters

Returns

Example Code

Properties

Usage Pattern

On this page