Automations

Manage automations (rules) that fire actions when metrics match conditions.

Command	Description
`automations create`	Create an automation.
`automations delete`	Delete an automation.
`automations get`	Get an automation by ID.
`automations list`	List automations.
`automations update`	Update an automation.

Flag	Type	Required	Description
`--description`	text	no	Human-readable description shown in the UI.
`--conditions`	text	yes	JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on `combine_type` (`all` vs `any`). Condition shape: Common scorer_type values: - `behavior` — Judge-scored behavior (name = behavior name, e.g. "Relevance") - `static` — Built-in metrics like "duration" (ms) or "llm_cost" (USD) - `prompt`/`custom` — Prompt or custom scorer by name - `span_attribute` — Arbitrary span attribute key (name = attribute key) - `error` — Span error condition
`--combine-type`	`all`, `any`	yes	—
`--actions`	text	no	JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass `"slack"` in `communication_methods` to use them.
`--cooldown-period`	text	no	JSON object describing the minimum wait between triggers. Omit to leave the cooldown unset; if provided, both `value` and `unit` are required. Shape: `{ "value": <number>, "unit": "seconds" \| "minutes" \| "hours" \| "days" }` Example: `{ "value": 15, "unit": "minutes" }` (at least 15 min between triggers)
`--trigger-frequency`	text	no	JSON object describing the rate-limit window. Omit to leave unset; if provided, all three fields are required. Shape: `{ "count": <number>, "period": <number>, "period_unit": "seconds" \| "minutes" \| "hours" \| "days" }` Example: `{ "count": 5, "period": 1, "period_unit": "hours" }` (max 5 triggers per 1 hour)
`-o`, `--output`	`yaml`, `json`	no	Output format.

Flag	Type	Required	Description
`-o`, `--output`	`yaml`, `json`	no	Output format.

Flag	Type	Required	Description
`-o`, `--output`	`yaml`, `json`	no	Output format.

Flag	Type	Required	Description
`-o`, `--output`	`table`, `yaml`, `json`	no	Output format.

Update an existing automation. All fields other than the IDs are optional — only supplied fields are applied. Use active: true/false to enable or disable without changing other fields. Requires the developer role.

judgment automations update [OPTIONS] <ORGANIZATION_ID> <PROJECT_ID> <RULE_ID>

Arguments

Name	Required
`ORGANIZATION_ID`	yes
`PROJECT_ID`	yes
`RULE_ID`	yes

Options

Flag	Type	Required	Description
`--name`	text	no	New name for the automation.
`--description`	text	no	New description for the automation.
`--conditions`	text	no	JSON array of rule conditions. Each condition references a named metric/scorer on the project and a comparison. Items are ANDed or ORed together based on `combine_type` (`all` vs `any`). Condition shape: Common scorer_type values: - `behavior` — Judge-scored behavior (name = behavior name, e.g. "Relevance") - `static` — Built-in metrics like "duration" (ms) or "llm_cost" (USD) - `prompt`/`custom` — Prompt or custom scorer by name - `span_attribute` — Arbitrary span attribute key (name = attribute key) - `error` — Span error condition
`--combine-type`	`all`, `any`	no	—
`--actions`	text	no	JSON object describing what happens when the automation fires. All top-level keys are optional — include only the actions you want configured. Shape: Slack notifications are configured per-organization in the Judgment UI; pass `"slack"` in `communication_methods` to use them.
`--active`	boolean	no	Enable (true) or disable (false) the automation without modifying other fields.
`--cooldown-period`	text	no	JSON 2-tuple `[period, unit]` describing the minimum wait between triggers. Omit to leave unchanged. Shape: `[<period:number>, <unit:"seconds"\|"minutes"\|"hours"\|"days">]` Example: `[15, "minutes"]` (at least 15 min between triggers)
`--trigger-frequency`	text	no	JSON 3-tuple `[count, period, unit]` describing the rate-limit window. Omit to leave unchanged. Shape: `[<max_trigger_count:number>, <period:number>, <unit:"seconds"\|"minutes"\|"hours"\|"days">]` Example: `[5, 1, "hours"]` (max 5 triggers per 1 hour)
`-o`, `--output`	`yaml`, `json`	no	Output format.

--conditions shape

{
  "metric": {
    "scorer_type": "behavior" | "judge" | "prompt" | "custom" | "static" | "span_attribute" | "error",
    "name": "<scorer or metric name>",
    "threshold": <number | string | null>?
  },
  "comparison": "lt" | "gt" | "eq" | "gte" | "lte" | "fails" | "succeeds" | "chooses" | "detected" | "equals" | "contains" | "exists"
}

--actions shape

{
  "notification": {
    "enabled": <bool>?,
    "communication_methods": ["email" | "slack" | "pagerduty"],
    "email_addresses": ["<addr>", ...]?,
    "pagerduty_config": {"routing_key":"<key>","severity":"critical"|"error"|"warning"|"info"}?
  }?,
  "dataset_addition": {
    "enabled": <bool>?,
    "dataset_name": "<dataset>",
    "metadata_fields": <any>?
  }?,
  "behavior_evaluation": {
    "enabled": <bool>?,
    "behavior_judge_names": ["<judge_name>", ...]
  }?
}

Automations

Commands

`automations create`

`automations delete`

`automations get`

`automations list`

`automations update`

On this page