Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview

Create judge

Create an Agent judge (wire value prompt), a Code judge (wire value custom), or a deterministic judge with categorical outputs and its initial production version. Deterministic judges require two outputs ordered match, then miss. This call fixes the judge type for its lifetime. Agent judges require prompt configuration; Code judges require a base64-encoded tar.gz bundle and Python entrypoint metadata.

POST/v1/projects/{projectId}/judges
Authorization
AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters
projectIdstringrequired
Request body
requiredapplication/json
judgeobjectrequired
Show properties
One of:
object
namestringrequired

Unique judge name within the project

min length 1
judgeDescriptionstring | null

Human-readable description shown in the UI

judgeTypestringrequired

Agent judge (wire value prompt)

initialVersionobjectrequired
Show properties
actionConfigobject | null
Show properties
Any of:
object
action_groupsobject[]required
Show properties
Array of object
idstringrequired
min length 1
namestringrequired

Action group display name; an empty string uses Action group 1, 2, etc.

enabledbooleanrequired
environment_namesstring[]required

Empty runs the group in every environment

whenobjectrequired

Judge result that runs the group. Categorical groups run when the result is any listed output name.

Show properties
One of:
object
kindstringrequired
object
kindstringrequired
valuebooleanrequired
object
kindstringrequired
valuesstring[]required
min items 1
object
kindstringrequired
operatorstringrequired
Allowed:ltlteeqgtegt
valuenumberrequired
frequencyobjectrequired
Show properties
countintegerrequired
min 1 · max 9007199254740991
window_secondsintegerrequired
min 1 · max 9007199254740991
cooldown_secondsintegerrequired
min 0 · max 9007199254740991
actionsobject[]required
Show properties
Array of object
One of:
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
urlstringrequired
min length 1
secretstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
email_addressstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
team_idstring
min length 1
channel_idstring
min length 1
messagestring
custom_attribute_keysstring[]
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
dataset_idstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
judge_idsstring[]required
min items 1
null
null
scoreTypestringrequired
outputsobject[]required
min items 2
Show properties
Array of object
namestringrequired
descriptionstringrequired
advancedSettingsobject
Show properties
online_environment_namesstring[] | nullrequired

null leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.

Show properties
Any of:
string[]
Array of string
string
null
null
online_evaluation_modestringrequired

continuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.

Allowed:continuouson_demand
online_sampling_ratenumberrequired

Percent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.

min 0 · max 100
online_span_triggersany | objectrequired
Show properties
Any of:
any
any
object
kindstringrequired
Allowed:andor
conditionsany[]required
online_session_scoringbooleanrequired
session_evaluation_count_enabledboolean | null
session_evaluation_idle_enabledboolean | null
session_evaluation_min_candidate_tracesnumber | null
session_evaluation_max_wait_minutesnumber | null
Show properties
Any of:
number
number
null
null
configobjectrequired
Show properties
promptstringrequired

Judge prompt template

min length 1
descriptionstring | null

Description stored on the initial Agent judge version

default: null
reasoningLevelstring

Reasoning level for the Agent judge runtime

default: "lite"
Allowed:litenormalhigh
minScorenumber
default: 0
maxScorenumber
default: 1
object
namestringrequired

Unique judge name within the project

min length 1
judgeDescriptionstring | null

Human-readable description shown in the UI

judgeTypestringrequired

Code judge (wire value custom)

initialVersionobjectrequired
Show properties
actionConfigobject | null
Show properties
Any of:
object
action_groupsobject[]required
Show properties
Array of object
idstringrequired
min length 1
namestringrequired

Action group display name; an empty string uses Action group 1, 2, etc.

enabledbooleanrequired
environment_namesstring[]required

Empty runs the group in every environment

whenobjectrequired

Judge result that runs the group. Categorical groups run when the result is any listed output name.

Show properties
One of:
object
kindstringrequired
object
kindstringrequired
valuebooleanrequired
object
kindstringrequired
valuesstring[]required
min items 1
object
kindstringrequired
operatorstringrequired
Allowed:ltlteeqgtegt
valuenumberrequired
frequencyobjectrequired
Show properties
countintegerrequired
min 1 · max 9007199254740991
window_secondsintegerrequired
min 1 · max 9007199254740991
cooldown_secondsintegerrequired
min 0 · max 9007199254740991
actionsobject[]required
Show properties
Array of object
One of:
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
urlstringrequired
min length 1
secretstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
email_addressstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
team_idstring
min length 1
channel_idstring
min length 1
messagestring
custom_attribute_keysstring[]
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
dataset_idstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
judge_idsstring[]required
min items 1
null
null
scoreTypestringrequired
outputsobject[]required
min items 2
Show properties
Array of object
namestringrequired
descriptionstringrequired
advancedSettingsobject
Show properties
online_environment_namesstring[] | nullrequired

null leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.

Show properties
Any of:
string[]
Array of string
string
null
null
online_evaluation_modestringrequired

continuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.

Allowed:continuouson_demand
online_sampling_ratenumberrequired

Percent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.

min 0 · max 100
online_span_triggersany | objectrequired
Show properties
Any of:
any
any
object
kindstringrequired
Allowed:andor
conditionsany[]required
online_session_scoringbooleanrequired
session_evaluation_count_enabledboolean | null
session_evaluation_idle_enabledboolean | null
session_evaluation_min_candidate_tracesnumber | null
session_evaluation_max_wait_minutesnumber | null
Show properties
Any of:
number
number
null
null
configobjectrequired
Show properties
bundleBase64stringrequired

Base64-encoded tar.gz bundle. Decoded size must be at most 1 MiB.

min length 1 · max length 1398104
entrypointPathstringrequired

Path to the Python entrypoint inside the tar.gz bundle

min length 1
requirementsPathstring

Optional requirements file path inside the bundle

min length 1
classNamestringrequired

Python scorer class name exported by the entrypoint

min length 1
scorerTypestring | null

Input format expected by the Code judge. Defaults to 'example'.

Show properties
Any of:
string
string
null
null
sdkVersioninteger

judgeval SDK major version targeted by the Code judge

max 9007199254740991 · default: 4
descriptionstring | null

Description stored on the Code judge version

object
namestringrequired
min length 1
judgeDescriptionstring | null
judgeTypestringrequired
initialVersionobjectrequired
Show properties
actionConfigobject | null
Show properties
Any of:
object
action_groupsobject[]required
Show properties
Array of object
idstringrequired
min length 1
namestringrequired

Action group display name; an empty string uses Action group 1, 2, etc.

enabledbooleanrequired
environment_namesstring[]required

Empty runs the group in every environment

whenobjectrequired

Judge result that runs the group. Categorical groups run when the result is any listed output name.

Show properties
One of:
object
kindstringrequired
object
kindstringrequired
valuebooleanrequired
object
kindstringrequired
valuesstring[]required
min items 1
object
kindstringrequired
operatorstringrequired
Allowed:ltlteeqgtegt
valuenumberrequired
frequencyobjectrequired
Show properties
countintegerrequired
min 1 · max 9007199254740991
window_secondsintegerrequired
min 1 · max 9007199254740991
cooldown_secondsintegerrequired
min 0 · max 9007199254740991
actionsobject[]required
Show properties
Array of object
One of:
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
urlstringrequired
min length 1
secretstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
email_addressstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
team_idstring
min length 1
channel_idstring
min length 1
messagestring
custom_attribute_keysstring[]
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
dataset_idstringrequired
min length 1
object
idstringrequired
min length 1
enabledbooleanrequired
typestringrequired
configobjectrequired
Show properties
judge_idsstring[]required
min items 1
null
null
scoreTypestringrequired
outputsobject[]required

Exactly two outputs: first for a match, second for a miss

min items 2 · max items 2
Show properties
Array of object
namestringrequired
descriptionstringrequired
advancedSettingsobject
Show properties
online_environment_namesstring[] | nullrequired

null leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.

Show properties
Any of:
string[]
Array of string
string
null
null
online_evaluation_modestringrequired

continuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.

Allowed:continuouson_demand
online_sampling_ratenumberrequired

Percent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.

min 0 · max 100
online_span_triggersany | objectrequired
Show properties
Any of:
any
any
object
kindstringrequired
Allowed:andor
conditionsany[]required
online_session_scoringbooleanrequired
session_evaluation_count_enabledboolean | null
session_evaluation_idle_enabledboolean | null
session_evaluation_min_candidate_tracesnumber | null
session_evaluation_max_wait_minutesnumber | null
Show properties
Any of:
number
number
null
null
configobjectrequired
Show properties
expressionanyrequired
Responses
200

Response for status 200

judge_idstringrequired
judge_version_idstringrequired
implementation_idstringrequired
Try it
Server
Authorization
Parameters
Bodyapplication/json
Request
curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/judges' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "judge": {
    "name": "string",
    "judgeDescription": "string",
    "judgeType": "prompt",
    "initialVersion": {
      "actionConfig": {
        "action_groups": [
          {
            "id": "string",
            "name": "string",
            "enabled": true,
            "environment_names": [
              "string"
            ],
            "when": {
              "kind": "any"
            },
            "frequency": {
              "count": 1,
              "window_seconds": 1
            },
            "cooldown_seconds": 0,
            "actions": [
              {
                "id": "string",
                "enabled": true,
                "type": "webhook",
                "config": {
                  "url": "string",
                  "secret": "string"
                }
              }
            ]
          }
        ]
      },
      "scoreType": "categorical",
      "outputs": [
        {
          "name": "string",
          "description": "string"
        },
        {
          "name": "string",
          "description": "string"
        }
      ],
      "advancedSettings": {
        "online_environment_names": [
          "string"
        ],
        "online_evaluation_mode": "continuous",
        "online_sampling_rate": 0,
        "online_span_triggers": {},
        "online_session_scoring": true,
        "session_evaluation_count_enabled": true,
        "session_evaluation_idle_enabled": true,
        "session_evaluation_min_candidate_traces": 0,
        "session_evaluation_max_wait_minutes": 1
      }
    },
    "config": {
      "prompt": "string",
      "description": null,
      "reasoningLevel": "lite",
      "minScore": 0,
      "maxScore": 1
    }
  }
}'
Response
{
  "judge_id": "string",
  "judge_version_id": "string",
  "implementation_id": "string"
}

Was this page helpful?