Prompt

The Prompt class provides an interface to easily create and version prompts that are stored on the Judgment platform. You can then dynamically retrieve any version of any prompt to be used in your application code.

Prompt's versioning system allows you to iterate on commits by creating multiple commits. You can also label prompt commits with tags that serve as a keywords to retrieve a specific version of a prompt.

from openai import OpenAI
from judgeval.prompt import Prompt

# Create a prompt. If the prompt name already exists, a new commit will be created
Prompt.create(
  project_name="default_project",
  name="Movie Recommendation Prompt",
  prompt="""Given this genre: {{genre}} and this decade: {{decade}},
  can you recommend a movie of that genre that was released in that decade?""",
  tags=["production"],
)

class MovieRecommendationAgent:
    def __init__(self):
        self.client = OpenAI()

    def get_recommendation(self, user_genre: str, user_decade: str):
        prompt = Prompt.get(
          project_name="default_project",
          name="Movie Recommendation Prompt",
          tag="production"
        )
        response = self.client.chat.completions.create(
            model="gpt-5",
            messages=[
              {
                "role": "user",
                "content": prompt.compile(genre=user_genre, decade=user_decade)
              }
            ]
        )
        return f"Response: {response.choices[0].message.content}"

if __name__ == "__main__":
    agent = MovieRecommendationAgent()
    agent.get_recommendation("horror", "90s")

Prompt.create()

Initialize a Prompt object.

Prompt.create(
  project_name: str,
  name: str,
  prompt: str,
  tags: Optional[List[str]] = None,
  judgment_api_key: Optional[str] = None,
  organization_id: Optional[str] = None
)

Parameters

Each tag is unique to one commit per prompt. Reusing a tag moves it from its previous commit to the new one.

project_namerequired:str

The name of the project to create a prompt in

namerequired:str

The name of the prompt, which is unique to each prompt. Specifying a name that already exists will create a new commit for that prompt

promptrequired:str

The prompt contents to save. Use {{ mustache notation }} for variables to specify parts of the prompt that can be dynamically replaced at runtime using Prompt.compile()

tags:Optional[List[str]]

If specified, the tags in this list will be attached to the created commit. Please read the warning in the Overview section for more details on tagging behavior

judgment_api_key:Optional[str]

Recommended - set using the JUDGMENT_API_KEY environment variable

organization_id:Optional[str]

Recommended - set using the JUDGMENT_ORG_ID environment variable

Example

create_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.create(
    project_name="default_project",
    name="Test Prompt",
    prompt="Recommend a movie in the {{genre}} genre",
    tags=["production"],
)

Prompt.get()

Retrieve the latest version of a Prompt that has already been created for the project.

Passing in a commit_id will retrieve the commit with that id for the prompt, if it exists.
Passing in a tag will retrieve the commit for the prompt with that tag attached, if it exists.

Prompt.get(
  project_name: str,
  name: str,
  commit_id: Optional[str] = None,
  tag: Optional[str] = None,
  judgment_api_key: Optional[str] = None,
  organization_id: Optional[str] = None
)

Parameters

project_namerequired:str

The name of the project to fetch the prompt from

namerequired:str

The name of the Prompt you would like to retrieve. Throws a JudgmentException if the prompt does not exist

commit_id:Optional[str]

The ID of the prompt commit you would like to retrieve. Mutually exclusive with tag. Throws a JudgmentException if the prompt commit does not exist

tag:Optional[str]

The tag associated with the prompt commit you would like to retrieve. Mutually exclusive with commit_id. Throws a JudgmentException if the prompt commit does not exist

judgment_api_key:Optional[str]

Recommended - set using the JUDGMENT_API_KEY environment variable

organization_id:Optional[str]

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

A Prompt instance. The Prompt instance will have the following metadata fields:

creator_first_name: The first name of the user who created the prompt
creator_last_name: The last name of the user who created the prompt
creator_email: The email of the user who created the prompt

Example

get_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.get(project_name="default_project", name="Test Prompt")

Prompt.tag()

Attach one or more tags to a specific version of a Prompt that has already been created for the project.

Prompt.tag(
  project_name: str,
  name: str,
  commit_id: Optional[str] = None,
  tags: Optional[List[str]] = None,
  judgment_api_key: Optional[str] = None,
  organization_id: Optional[str] = None
)

Parameters

Each tag is unique to one commit per prompt. Reusing a tag moves it from its previous commit to the new one.

project_namerequired:str

The name of the project that the prompt you would like to tag is in

namerequired:str

The name of the Prompt you would like to tag. Throws a JudgmentException if the prompt does not exist

commit_id:Optional[str]

The ID of the prompt commit you would like to attach tag(s) to. Throws a JudgmentException if the prompt commit does not exist

tags:Optional[List[str]]

The tag(s) you would like to add to the prompt commit. Throws a JudgmentException if this list is empty

judgment_api_key:Optional[str]

Recommended - set using the JUDGMENT_API_KEY environment variable

organization_id:Optional[str]

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

A str representing the commit ID which the tag(s) were attached to.

Example

tag_prompt.py

from judgeval.prompt import Prompt

commit_id = Prompt.tag(
    project_name="default_project",
    name="Test Prompt",
    commit_id="some_commit_id",
    tags=["production", "v1.2"],
)

Prompt.untag()

Remove one or more tags attached to any of the versions of a Prompt that has already been created for the project.

Prompt.untag(
  project_name: str,
  name: str,
  tags: Optional[List[str]] = None,
  judgment_api_key: Optional[str] = None,
  organization_id: Optional[str] = None
)

Parameters

project_namerequired:str

The name of the project that the prompt you would like to untag is in

namerequired:str

The name of the Prompt you would like to untag. Throws a JudgmentException if the prompt does not exist

tags:Optional[List[str]]

The tag(s) you would like to remove from the prompt. Throws a JudgmentException if this list is empty or if the prompt does not have any of the provided tags

judgment_api_key:Optional[str]

Recommended - set using the JUDGMENT_API_KEY environment variable

organization_id:Optional[str]

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

A List[str] representing the commit IDs which the tag(s) were removed from.

Example

untag_prompt.py

from judgeval.prompt import Prompt

prompt = Prompt.untag(
    project_name="default_project",
    name="Test Prompt",
    tags=["production"],
)

Prompt.list()

Retrieve all the versions of a Prompt that has already been created for the project.

Prompt.list(
  project_name: str,
  name: str,
  judgment_api_key: Optional[str] = None,
  organization_id: Optional[str] = None
)

Parameters

project_namerequired:str

The name of the project to fetch the prompt from

namerequired:str

The name of the Prompt whose versions you would like to retrieve. If the prompt does not exist, an empty list will be returned

judgment_api_key:Optional[str]

Recommended - set using the JUDGMENT_API_KEY environment variable

organization_id:Optional[str]

Recommended - set using the JUDGMENT_ORG_ID environment variable

Returns

A List[Prompt] representing the versions of the prompt. Each Prompt in the list will have the following metadata fields:

creator_first_name: The first name of the user who created the prompt
creator_last_name: The last name of the user who created the prompt
creator_email: The email of the user who created the prompt

Example

list_prompts.py

from judgeval.prompt import Prompt

prompts = Prompt.list(project_name="default_project", name="Test Prompt")

Return Types

Prompt

Class instances returned by Prompt.create(), Prompt.get(), and Prompt.list(), that provide details about a specific prompt version and additional methods for prompt management.

The Prompt object contains the following properties:

name:str

Name of the prompt, which is unique between prompts

prompt:str

Contents of the prompt

created_at:str

Timestamp of when the prompt commit was created

tags:List[str]

Tags attached to the prompt commit

commit_id:str

ID of the retrieved or created commit

parent_commit_id:Union[str, None]

Parent of the current commit

metadata:Dict[str, str]

Metadata about the prompt. This could include the first name and last name of the user who created the prompt. For example:

{
  "creator_first_name": "John",
  "creator_last_name": "Doe",
  "creator_email": "john.doe@example.com",
}

Prompt

On this page