Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

judgment automations

Manage automations (rules) that fire actions when metrics match conditions.

Manage automations (rules) that fire actions when metrics match conditions.

Commands

Command Description
automations create Create an automation.
automations delete Delete an automation.
automations get Get an automation by ID.
automations list List automations.
automations update Update an automation.

automations create

Create an automation.

Create an automation (rule) in a project. An automation watches behavior/latency/cost metrics and fires actions when its conditions match. Requires the developer role.

judgment automations create [OPTIONS] [[[ORG_ID] PROJECT_ID] NAME...]

Arguments

Name Required
[[ORG_ID] PROJECT_ID] NAME no

Options

Flag Type Required Description
–organization-id, –org-id text no Organization ID. Defaults to JUDGMENT_ORG_ID or saved context.
–organization, –org text no Organization name to resolve.
–project-id text no Project ID. Defaults to JUDGMENT_PROJECT_ID or saved context.
–project text no Project name to resolve.
–description text no Human-readable description shown in the UI.
–conditions text yes JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on combine_type (all vs any). Condition shape: Common scorer_type values: - behavior — Judge-scored behavior (name = behavior name, e.g. “Relevance”) - static — Built-in metrics like “duration” (ms) or “llm_cost” (USD) - prompt/custom — Prompt or custom scorer by name - span_attribute — Arbitrary span attribute key (name = attribute key) - error — Span error condition
–combine-type all, any yes —
–actions text no JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass "slack" in communication_methods to use them.
–cooldown-period text no JSON object describing the minimum wait between triggers. Omit to leave the cooldown unset; if provided, both value and unit are required. Shape: { "value": <number>, "unit": "seconds" | "minutes" | "hours" | "days" } Example: { "value": 15, "unit": "minutes" } (at least 15 min between triggers)
–trigger-frequency text no JSON object describing the rate-limit window. Omit to leave unset; if provided, all three fields are required. Shape: { "count": <number>, "period": <number>, "period_unit": "seconds" | "minutes" | "hours" | "days" } Example: { "count": 5, "period": 1, "period_unit": "hours" } (max 5 triggers per 1 hour)
-o, –output yaml, json no Output format.

--conditions shape

{
  "metric": {
    "scorer_type": "behavior" | "judge" | "prompt" | "custom" | "static" | "span_attribute" | "error",
    "name": "<scorer or metric name>",
    "threshold": <number | string | null>?
  },
  "comparison": "lt" | "gt" | "eq" | "gte" | "lte" | "fails" | "succeeds" | "chooses" | "detected" | "equals" | "contains" | "exists"
}

--actions shape

{
  "notification": {
    "enabled": <bool>?,
    "communication_methods": ["email" | "slack" | "pagerduty"],
    "email_addresses": ["<addr>", ...]?,
    "pagerduty_config": {"routing_key":"<key>","severity":"critical"|"error"|"warning"|"info"}?
  }?,
  "dataset_addition": {
    "enabled": <bool>?,
    "dataset_name": "<dataset>",
    "metadata_fields": <any>?
  }?,
  "behavior_evaluation": {
    "enabled": <bool>?,
    "behavior_judge_names": ["<judge_name>", ...]
  }?
}

automations delete

Delete an automation.

Delete an automation. Requires the admin role.

judgment automations delete [OPTIONS] [[[ORG_ID] PROJECT_ID] RULE_ID...]

Arguments

Name Required
[[ORG_ID] PROJECT_ID] RULE_ID no

Options

Flag Type Required Description
–organization-id, –org-id text no Organization ID. Defaults to JUDGMENT_ORG_ID or saved context.
–organization, –org text no Organization name to resolve.
–project-id text no Project ID. Defaults to JUDGMENT_PROJECT_ID or saved context.
–project text no Project name to resolve.
-o, –output yaml, json no Output format.

automations get

Get an automation by ID.

judgment automations get [OPTIONS] [[[ORG_ID] PROJECT_ID] RULE_ID...]

Arguments

Name Required
[[ORG_ID] PROJECT_ID] RULE_ID no

Options

Flag Type Required Description
–organization-id, –org-id text no Organization ID. Defaults to JUDGMENT_ORG_ID or saved context.
–organization, –org text no Organization name to resolve.
–project-id text no Project ID. Defaults to JUDGMENT_PROJECT_ID or saved context.
–project text no Project name to resolve.
-o, –output yaml, json no Output format.

automations list

List automations.

judgment automations list [OPTIONS] [[[ORG_ID] PROJECT_ID]...]

Arguments

Name Required
[[ORG_ID] PROJECT_ID] no

Options

Flag Type Required Description
–organization-id, –org-id text no Organization ID. Defaults to JUDGMENT_ORG_ID or saved context.
–organization, –org text no Organization name to resolve.
–project-id text no Project ID. Defaults to JUDGMENT_PROJECT_ID or saved context.
–project text no Project name to resolve.
-o, –output table, yaml, json no Output format.

automations update

Update an automation.

Update an existing automation. All fields other than the IDs are optional — only supplied fields are applied. Use active: true/false to enable or disable without changing other fields. Requires the developer role.

judgment automations update [OPTIONS] [[[ORG_ID] PROJECT_ID] RULE_ID...]

Arguments

Name Required
[[ORG_ID] PROJECT_ID] RULE_ID no

Options

Flag Type Required Description
–organization-id, –org-id text no Organization ID. Defaults to JUDGMENT_ORG_ID or saved context.
–organization, –org text no Organization name to resolve.
–project-id text no Project ID. Defaults to JUDGMENT_PROJECT_ID or saved context.
–project text no Project name to resolve.
–name text no New name for the automation.
–description text no New description for the automation.
–conditions text no JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on combine_type (all vs any). Condition shape: Common scorer_type values: - behavior — Judge-scored behavior (name = behavior name, e.g. “Relevance”) - static — Built-in metrics like “duration” (ms) or “llm_cost” (USD) - prompt/custom — Prompt or custom scorer by name - span_attribute — Arbitrary span attribute key (name = attribute key) - error — Span error condition
–combine-type all, any no —
–actions text no JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass "slack" in communication_methods to use them.
–active boolean no Enable (true) or disable (false) the automation without modifying other fields.
–cooldown-period text no JSON 2-tuple [period, unit] describing the minimum wait between triggers. Omit to leave unchanged. Shape: [&lt;period:number&gt;, &lt;unit:"seconds"|"minutes"|"hours"|"days"&gt;] Example: [15, "minutes"] (at least 15 min between triggers)
–trigger-frequency text no JSON 3-tuple [count, period, unit] describing the rate-limit window. Omit to leave unchanged. Shape: [&lt;max_trigger_count:number&gt;, &lt;period:number&gt;, &lt;unit:"seconds"|"minutes"|"hours"|"days"&gt;] Example: [5, 1, "hours"] (max 5 triggers per 1 hour)
-o, –output yaml, json no Output format.

--conditions shape

{
  "metric": {
    "scorer_type": "behavior" | "judge" | "prompt" | "custom" | "static" | "span_attribute" | "error",
    "name": "<scorer or metric name>",
    "threshold": <number | string | null>?
  },
  "comparison": "lt" | "gt" | "eq" | "gte" | "lte" | "fails" | "succeeds" | "chooses" | "detected" | "equals" | "contains" | "exists"
}

--actions shape

{
  "notification": {
    "enabled": <bool>?,
    "communication_methods": ["email" | "slack" | "pagerduty"],
    "email_addresses": ["<addr>", ...]?,
    "pagerduty_config": {"routing_key":"<key>","severity":"critical"|"error"|"warning"|"info"}?
  }?,
  "dataset_addition": {
    "enabled": <bool>?,
    "dataset_name": "<dataset>",
    "metadata_fields": <any>?
  }?,
  "behavior_evaluation": {
    "enabled": <bool>?,
    "behavior_judge_names": ["<judge_name>", ...]
  }?
}

Was this page helpful?