Update judge
Update a judge description and/or create one complete immutable judge version. A new version requires an explicit target version, the existing judge and score types, a complete implementation config, outputs, and monitoring-settings snapshot. Categorical outputs may reference existing labels by ID or define new labels. Code judge versions take a base64-encoded tar.gz bundle, or a sourceVersion whose stored code to reuse. Alternatively, newVersionFromSource copies sourceVersion and changes only the fields you set. Set promoteToProduction to make the new version production; latest is inferred automatically. Judge and score types cannot change.
/v1/projects/{projectId}/judges/{judgeId}AuthorizationPersonal API key · headerrequiredYour personal API key from Settings > Personal API Keys, sent as Bearer <key>.
X-Organization-IdOrganization ID · headerrequiredThe organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.
projectIdstringrequiredjudgeIdstring<uuid>requiredThe judge ID
application/jsonupdateobjectrequiredShow propertiesHide properties
judgeDescriptionstring | nullUpdated human-readable judge description
newVersionobjectComplete immutable version to create
Show propertiesHide properties
actionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredAgent judge (wire value prompt)
configobjectrequiredShow propertiesHide properties
promptstringrequiredComplete prompt for the new version
descriptionstring | nullrequiredDescription stored on the new Agent judge version
reasoningLevelstringrequiredReasoning level for the new Agent judge version
litenormalhighminScorenumberrequiredMinimum numeric score for the new version
maxScorenumberrequiredMaximum numeric score for the new version
scoreTypestringrequiredactionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredAgent judge (wire value prompt)
configobjectrequiredShow propertiesHide properties
promptstringrequiredComplete prompt for the new version
descriptionstring | nullrequiredDescription stored on the new Agent judge version
reasoningLevelstringrequiredReasoning level for the new Agent judge version
litenormalhighminScorenumberrequiredMinimum numeric score for the new version
maxScorenumberrequiredMaximum numeric score for the new version
scoreTypestringrequiredactionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredAgent judge (wire value prompt)
configobjectrequiredShow propertiesHide properties
promptstringrequiredComplete prompt for the new version
descriptionstring | nullrequiredDescription stored on the new Agent judge version
reasoningLevelstringrequiredReasoning level for the new Agent judge version
litenormalhighminScorenumberrequiredMinimum numeric score for the new version
maxScorenumberrequiredMaximum numeric score for the new version
scoreTypestringrequiredoutputsobject[]requiredComplete categorical output set for the new version
Show propertiesHide properties
objectbehaviorIdstringrequiredAn existing label ID. The field name remains behaviorId.
namestringrequiredName of the label to create
descriptionstringrequiredDescription of the label to create
categoryIdsstring[]Label category IDs from the same project as the judge
actionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredCode judge (wire value custom)
configobjectrequiredShow propertiesHide properties
bundleBase64stringrequiredBase64-encoded tar.gz bundle. Decoded size must be at most 1 MiB.
entrypointPathstringrequiredPath to the Python entrypoint inside the tar.gz bundle
requirementsPathstringOptional requirements file path inside the bundle
classNamestringrequiredPython scorer class name exported by the entrypoint
scorerTypestring | nullInput format expected by the Code judge. Defaults to 'example'.
Show propertiesHide properties
stringnullsdkVersionintegerjudgeval SDK major version targeted by the Code judge
descriptionstring | nullDescription stored on the Code judge version
sourceVersionobjectrequiredExisting version whose stored code the new version reuses
Show propertiesHide properties
majorintegerrequiredminorintegerrequiredscoreTypestringrequiredactionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredCode judge (wire value custom)
configobjectrequiredShow propertiesHide properties
bundleBase64stringrequiredBase64-encoded tar.gz bundle. Decoded size must be at most 1 MiB.
entrypointPathstringrequiredPath to the Python entrypoint inside the tar.gz bundle
requirementsPathstringOptional requirements file path inside the bundle
classNamestringrequiredPython scorer class name exported by the entrypoint
scorerTypestring | nullInput format expected by the Code judge. Defaults to 'example'.
Show propertiesHide properties
stringnullsdkVersionintegerjudgeval SDK major version targeted by the Code judge
descriptionstring | nullDescription stored on the Code judge version
sourceVersionobjectrequiredExisting version whose stored code the new version reuses
Show propertiesHide properties
majorintegerrequiredminorintegerrequiredscoreTypestringrequiredactionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredCode judge (wire value custom)
configobjectrequiredShow propertiesHide properties
bundleBase64stringrequiredBase64-encoded tar.gz bundle. Decoded size must be at most 1 MiB.
entrypointPathstringrequiredPath to the Python entrypoint inside the tar.gz bundle
requirementsPathstringOptional requirements file path inside the bundle
classNamestringrequiredPython scorer class name exported by the entrypoint
scorerTypestring | nullInput format expected by the Code judge. Defaults to 'example'.
Show propertiesHide properties
stringnullsdkVersionintegerjudgeval SDK major version targeted by the Code judge
descriptionstring | nullDescription stored on the Code judge version
sourceVersionobjectrequiredExisting version whose stored code the new version reuses
Show propertiesHide properties
majorintegerrequiredminorintegerrequiredscoreTypestringrequiredoutputsobject[]requiredComplete categorical output set for the new version
Show propertiesHide properties
objectbehaviorIdstringrequiredAn existing label ID. The field name remains behaviorId.
namestringrequiredName of the label to create
descriptionstringrequiredDescription of the label to create
categoryIdsstring[]Label category IDs from the same project as the judge
actionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredconfigobjectrequiredShow propertiesHide properties
expressionanyrequiredscoreTypestringrequiredactionConfigobject | nullOmit to preserve the latest version’s action groups; null clears them.
Show propertiesHide properties
action_groupsobject[]requiredShow propertiesHide properties
objectidstringrequirednamestringrequiredAction group display name; an empty string uses Action group 1, 2, etc.
enabledbooleanrequiredenvironment_namesstring[]requiredEmpty runs the group in every environment
whenobjectrequiredJudge result that runs the group. Categorical groups run when the result is any listed output name.
Show propertiesHide properties
kindstringrequiredkindstringrequiredvaluebooleanrequiredkindstringrequiredvaluesstring[]requiredkindstringrequiredoperatorstringrequiredltlteeqgtegtvaluenumberrequiredfrequencyobjectrequiredShow propertiesHide properties
countintegerrequiredwindow_secondsintegerrequiredcooldown_secondsintegerrequiredactionsobject[]requiredShow propertiesHide properties
objectidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
urlstringrequiredsecretstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
email_addressstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
team_idstringchannel_idstringmessagestringcustom_attribute_keysstring[]idstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
dataset_idstringrequiredidstringrequiredenabledbooleanrequiredtypestringrequiredconfigobjectrequiredShow propertiesHide properties
judge_idsstring[]requirednulltargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectrequiredComplete monitoring-settings snapshot for the new version
Show propertiesHide properties
online_environment_namesstring[] | nullrequirednull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringrequiredcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberrequiredPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectrequiredShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleanrequiredsession_evaluation_count_enabledboolean | nullrequiredsession_evaluation_idle_enabledboolean | nullrequiredsession_evaluation_min_candidate_tracesnumber | nullrequiredsession_evaluation_max_wait_minutesnumber | nullrequiredShow propertiesHide properties
numbernulljudgeTypestringrequiredconfigobjectrequiredShow propertiesHide properties
expressionanyrequiredscoreTypestringrequiredoutputsobject[]requiredExactly two outputs: first for a match, second for a miss
Show propertiesHide properties
objectbehaviorIdstringrequiredAn existing label ID. The field name remains behaviorId.
namestringrequiredName of the label to create
descriptionstringrequiredDescription of the label to create
categoryIdsstring[]Label category IDs from the same project as the judge
newVersionFromSourceobjectNew version that copies sourceVersion and changes only the fields you set. Arrays replace.
Show propertiesHide properties
sourceVersionobjectrequiredExisting version to copy. Fields you omit keep its values.
Show propertiesHide properties
majorintegerrequiredminorintegerrequiredtargetVersionobjectrequiredShow propertiesHide properties
majorintegerrequiredminorintegerrequiredpromoteToProductionbooleanrequiredWhether the new version becomes the production version
advancedSettingsobjectOnly the settings to change. Set online_evaluation_mode to 'on_demand' and promote to stop continuous scoring.
Show propertiesHide properties
online_environment_namesstring[] | nullnull leaves on-demand evaluation unconfigured; online evaluation requires [] for All or selected names.
Show propertiesHide properties
stringstringnullonline_evaluation_modestringcontinuous checks new traces; on_demand checks only traces evaluations are requested on. Both apply environments, run conditions, and sampling.
continuouson_demandonline_sampling_ratenumberPercent of matching traces (or sessions, with session scoring on) scored, including evaluations requested on traces.
online_span_triggersany | objectShow propertiesHide properties
anykindstringrequiredandorconditionsany[]requiredonline_session_scoringbooleansession_evaluation_count_enabledboolean | nullsession_evaluation_idle_enabledboolean | nullsession_evaluation_min_candidate_tracesnumber | nullsession_evaluation_max_wait_minutesnumber | nullShow propertiesHide properties
numbernulloutputsobject[]Categorical judges only. Replaces the complete output set. Changing outputs affects labels already applied and test-suite checks that expect them; check them first.
Show propertiesHide properties
objectbehaviorIdstringrequiredAn existing label ID. The field name remains behaviorId.
namestringrequiredName of the label to create
descriptionstringrequiredDescription of the label to create
categoryIdsstring[]Label category IDs from the same project as the judge
configobjectOnly the fields to change, from the judge type's fields: prompt, description, reasoningLevel, minScore, maxScore for Agent judges; expression for deterministic judges. Code judges: omit to reuse the stored bundle; to upload new code, set bundleBase64, entrypointPath, and className.
Show propertiesHide properties
promptstringComplete prompt for the new version
descriptionstring | nullDescription stored on the new version
reasoningLevelstringReasoning level for the new Agent judge version
litenormalhighminScorenumberMinimum numeric score for the new version
maxScorenumberMaximum numeric score for the new version
bundleBase64stringBase64-encoded tar.gz bundle. Decoded size must be at most 1 MiB.
entrypointPathstringPath to the Python entrypoint inside the tar.gz bundle
requirementsPathstringOptional requirements file path inside the bundle
classNamestringPython scorer class name exported by the entrypoint
scorerTypestringInput format expected by the Code judge. Omit to keep the source version's.
exampletracesdkVersionintegerjudgeval SDK major version targeted by the Code judge
expressionanyResponse for status 200
judge_idstringrequiredjudge_version_idstring | nullrequiredimplementation_idstring | nullrequired