Automations

Manage automations (rules) that fire actions when metrics match conditions.

Command	Description
`automations create`	Create an automation.
`automations delete`	Delete an automation.
`automations get`	Get an automation by ID.
`automations list`	List automations.
`automations update`	Update an automation.

Flag	Type	Required	Description
`--description`	text	no	Human-readable description shown in the UI.
`--conditions`	text	yes	JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on `combine_type` (`all` vs `any`). Condition shape: Common scorer_type values: - `behavior` — Judge-scored behavior (name = behavior name, e.g. "Relevance") - `static` — Built-in metrics like "duration" (ms) or "llm_cost" (USD) - `prompt`/`custom` — Prompt or custom scorer by name - `span_attribute` — Arbitrary span attribute key (name = attribute key) - `error` — Span error condition
`--combine-type`	`all`, `any`	yes	—
`--actions`	text	no	JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass `"slack"` in `communication_methods` to use them.
`--cooldown-period`	number	no	Cooldown duration as a plain number, paired with `cooldown_period_unit`. Example: 15 with unit `"minutes"`.
`--cooldown-period-unit`	`seconds`, `minutes`, `hours`, `days`	no	—
`--trigger-frequency-count`	number	no	Maximum triggers allowed within the trigger-frequency window.
`--trigger-frequency-period`	number	no	Window length for the trigger-frequency limit, paired with `trigger_frequency_period_unit`.
`--trigger-frequency-period-unit`	`seconds`, `minutes`, `hours`, `days`	no	—

Update an existing automation. All fields other than the IDs are optional — only supplied fields are applied. Use active: true/false to enable or disable without changing other fields. Requires the developer role.

judgment automations update [OPTIONS] <PROJECT_ID> <RULE_ID>

Arguments

Name	Required
`PROJECT_ID`	yes
`RULE_ID`	yes

Options

Flag	Type	Required	Description
`--name`	text	no	New name for the automation.
`--description`	text	no	New description for the automation.
`--conditions`	text	no	JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on `combine_type` (`all` vs `any`). Condition shape: Common scorer_type values: - `behavior` — Judge-scored behavior (name = behavior name, e.g. "Relevance") - `static` — Built-in metrics like "duration" (ms) or "llm_cost" (USD) - `prompt`/`custom` — Prompt or custom scorer by name - `span_attribute` — Arbitrary span attribute key (name = attribute key) - `error` — Span error condition
`--combine-type`	`all`, `any`	no	—
`--actions`	text	no	JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass `"slack"` in `communication_methods` to use them.
`--active`	boolean	no	Enable (true) or disable (false) the automation without modifying other fields.
`--cooldown-period`	text	no	JSON 2-tuple `[period, unit]` describing the minimum wait between triggers. Omit to leave unchanged. Shape: `[<period:number>, <unit:"seconds"\|"minutes"\|"hours"\|"days">]` Example: `[15, "minutes"]` (at least 15 min between triggers)
`--trigger-frequency`	text	no	JSON 3-tuple `[count, period, unit]` describing the rate-limit window. Omit to leave unchanged. Shape: `[<max_trigger_count:number>, <period:number>, <unit:"seconds"\|"minutes"\|"hours"\|"days">]` Example: `[5, 1, "hours"]` (max 5 triggers per 1 hour)

--conditions shape

{
  "metric": {
    "scorer_type": "behavior" | "judge" | "prompt" | "custom" | "static" | "span_attribute" | "error",
    "name": "<scorer or metric name>",
    "threshold": <number | string | null>?
  },
  "comparison": "lt" | "gt" | "eq" | "gte" | "lte" | "fails" | "succeeds" | "chooses" | "detected" | "equals" | "contains" | "exists"
}

--actions shape

{
  "notification": {
    "enabled": <bool>?,
    "communication_methods": ["email" | "slack" | "pagerduty"],
    "email_addresses": ["<addr>", ...]?,
    "pagerduty_config": {"routing_key":"<key>","severity":"critical"|"error"|"warning"|"info"}?
  }?,
  "dataset_addition": {
    "enabled": <bool>?,
    "dataset_name": "<dataset>",
    "metadata_fields": <any>?
  }?,
  "behavior_evaluation": {
    "enabled": <bool>?,
    "behavior_judge_names": ["<judge_name>", ...]
  }?
}

Automations

Commands

`automations create`

`automations delete`

`automations get`

`automations list`

`automations update`

On this page