--- title: "Configure automations and alerts" description: "Run actions when a completed trace matches behavior or trace conditions." sidebar: label: "Configure automations and alerts" seo: title: "Configure Automations and Alerts | Judgment How-to" description: "Create per-trace automation conditions, choose notification or routing actions, control frequency and cooldown, and verify the result." --- Automations evaluate conditions against each completed trace. When a trace matches, Judgment can notify a team, add the trace to a dataset, or run a behavior evaluation. This guide creates one automation and verifies its action. > **Automations evaluate individual traces** > > A condition does not calculate a rate, trend, or week-over-week change. Use an > automation for “act when this trace matches.” Use behaviors for trends, or > Issues for scheduled analysis. ## Prerequisites - An instrumented agent sending completed traces to the target project - A known condition you can trigger safely in a test trace - Access to the notification destination or dataset used by the action For a behavior condition, first [create and monitor the behavior](/documentation/judges/behavior-setup). For a trace condition, confirm that the required duration, cost, exception, or [span attribute](/documentation/tracing/attributes) is present on incoming traces. ## 1. Create the automation Open **Monitoring**, select the **Automations** tab, and then select **New Automation**. Enter a name that states the event and response, such as `Unsafe response to Slack`. Add a description when the threshold or owner would not otherwise be clear. Leave the automation enabled so you can verify it with new traffic. ## 2. Add conditions Under **Trigger on**, select **Add trigger** and choose one of the supported condition families: | Condition | Supported comparison | | -------------------- | -------------------------------------------------------- | | Duration or LLM cost | Greater than, less than, equal to, or inclusive variants | | Exception | Exists, equals, or contains | | Span attribute | Exists, equals, or contains | | Behavior | The selected binary or categorical behavior is triggered | For example, choose **Behaviors > Behavior triggered**, then select the `unsafe` behavior beneath its owning judge. Add more conditions when necessary. Choose **AND conditions** when every condition must match the same trace, or **OR conditions** when any one is sufficient. Keep the first verification to one unambiguous condition. ## 3. Choose at least one action Under **Actions**, select **Add action**. An automation can run more than one action: | Action | Configuration | | ------------------ | ------------------------------------------------------- | | Send email | One or more recipient addresses | | Notify on Slack | One or more connected Slack channels | | Alert on PagerDuty | Integration key and severity | | Send webhook | HTTPS endpoint and generated signing secret | | Add to dataset | Existing dataset or a new dataset | | Run behavior | One or more behaviors to evaluate on the matching trace | Choose an action you can observe directly. For a first test, adding to a small dataset or sending to a test notification destination makes verification straightforward. > **Info** > > Webhook actions are in limited availability. Contact Judgment to enable them > for your organization. ## 4. Set frequency and cooldown limits **Run actions only if conditions matched at least** controls action frequency. Its default is `1 time in the last 1 second`, so each matching trace can run the actions. **Once actions are triggered, wait** controls the cooldown. Its default is `0 seconds`, so there is no delay before the next eligible action. Use frequency to require several matching traces within a rolling window. Use cooldown to prevent repeated notifications or actions after the threshold is met. Both support seconds, minutes, hours, and days. ## 5. Save and verify Save the automation, then send one controlled trace that satisfies its condition. Wait for the root span to complete and verify both sides: - In **Logs > Traces**, the trace appears with the automation in its **Automations** column. - The configured action happened: the message arrived, the trace appeared in the dataset, or the selected behavior produced a result. If the condition matched but the action did not run, check whether the frequency threshold has been reached and whether the automation is still in its cooldown period. Also confirm that the automation is enabled and that the trace completed after you saved it. ## Condition and action boundaries - Conditions and actions operate on the same finalized trace. - A behavior condition uses a binary or categorical signal defined under its owning judge. Numeric judge outputs are not behaviors. - **Run behavior** can evaluate selected behaviors after a trace first matches a cheaper trace condition, such as a span attribute. - **Add to dataset** preserves matching traces for later review and offline testing. - Email, Slack, PagerDuty, and webhook actions are subject to the same frequency and cooldown limits. For the complete behavior model, see [Judges, results, and behaviors](/documentation/judges). For trace condition semantics, see [Trace conventions](/documentation/reference/trace-conventions). ## Webhook reference Webhook setup and payload details are reference material for teams that choose the webhook action. They do not change the automation setup path above. ### Configure the endpoint 1. Add the **Send webhook** action. Judgment generates a signing secret for the endpoint. Copy it into your receiver's secret store. 2. Enter the URL that will receive deliveries. The URL must use HTTPS on the default port, resolve through a public DNS hostname, and contain no credentials. IP literals, `localhost`, and `.local` names are rejected. 3. Select **Send test**. Judgment sends a synthetic event with the same shape as a real one and sets `test` to `true`. Confirm that your verification path accepts it before saving the automation. Judgment implements the [Standard Webhooks](https://www.standardwebhooks.com) specification, so you can verify deliveries with any conforming library. ### Request format Every delivery is a `POST` with a JSON body and these headers: | Header | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------- | | `content-type` | Always `application/json`. | | `user-agent` | `JudgmentLabs-Webhooks/1.0.0` | | `webhook-id` | Unique delivery identifier. Retries of the same delivery reuse this value, and it matches the `id` field in the body. | | `webhook-timestamp` | Unix timestamp in seconds at which the request was signed. | | `webhook-signature` | Space-separated list of versioned signatures, each formatted `v1,`. | ### Event payload Version `1` defines one event type, `automation.rule.triggered`: ```json { "id": "8f2b7ac1-4d1e-4c3a-9f70-6f6e1b2c9a05", "type": "automation.rule.triggered", "version": "1", "test": false, "created_at": "2026-07-20T18:42:11.204Z", "organization_id": "0b0f3a2a-3a6c-4a4d-9c6a-2f1f0a6d5e11", "project_id": "3c9d1f77-2a44-4f2e-bb84-0f2c9c3a7d10", "data": { "automation": { "id": "b6d4d1e2-9f3b-4a71-8b1f-2c0f7a5e4c33", "name": "Slow checkout agent" }, "trace": { "id": "9a7c1f0e2b3d4c5a6e7f8091a2b3c4d5", "started_at": "2026-07-20T18:42:09.980Z", "duration_ms": 1250, "llm_cost_usd": 0.0025, "has_error": false }, "condition_evaluations": [ { "metric": "duration", "unit": "seconds", "comparison": "gt", "threshold": 1, "observed_value": 1.25, "matched": true } ] } } ``` | Field | Type | Description | | ---------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------ | | `id` | `string` | UUID identifying the delivery. Stable across retries. | | `type` | `string` | Event type. Currently always `automation.rule.triggered`. | | `version` | `string` | Contract version of the packet. | | `test` | `boolean` | `true` for a synthetic delivery sent from the configuration screen, `false` for a real automation run. | | `created_at` | `string` | ISO 8601 timestamp with a UTC or numeric offset. | | `organization_id` | `string` | Organization that owns the automation. | | `project_id` | `string` | Project the automation belongs to. | | `data.automation.id` | `string` | Automation identifier. | | `data.automation.name` | `string` | Automation name at the time of the trigger. | | `data.trace.id` | `string` | Trace that matched the automation. | | `data.trace.started_at` | `string` | ISO 8601 start time of the trace. | | `data.trace.duration_ms` | `number \| null` | Trace duration in milliseconds. | | `data.trace.llm_cost_usd` | `number \| null` | LLM cost of the trace in USD. | | `data.trace.has_error` | `boolean` | Whether the trace recorded an error. | | `data.condition_evaluations` | `array` | One entry per evaluated condition. | Each entry in `condition_evaluations` describes one condition: | Field | Type | Description | | ---------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `metric` | `string` | `duration`, `llm_cost`, `error` for an exception condition, the span-attribute key, or the owning `judge_name` for a judge or behavior condition. | | `unit` | `"seconds" \| "usd" \| null` | Unit of `threshold` and `observed_value`. `null` for unitless metrics. | | `comparison` | `string` | One of `lt`, `gt`, `eq`, `gte`, `lte`, `fails`, `succeeds`, `chooses`, `detected`, `equals`, `contains`, `exists`. | | `threshold` | `number \| string \| null` | Configured threshold, when the comparison takes one. For a categorical behavior condition, this is the selected behavior value. | | `observed_value` | `number \| string \| null` | Value measured on the trace. For a categorical behavior condition, this is the choice returned by the judge. | | `matched` | `boolean` | Whether this individual condition was satisfied. | Conditions skipped during evaluation are omitted. Raw trace content, span attributes, and error messages are never included in the payload. Use `data.trace.id` to fetch those from the API. A breaking payload change ships as a new `version`. For a categorical behavior condition, `metric` does not contain the behavior value. It contains the owning judge's `judge_name`; `threshold` contains the selected behavior value, and `observed_value` contains the judge's returned choice. ### Verify the signature The `webhook-signature` header contains a base64-encoded HMAC-SHA256 signature of `{webhook-id}.{webhook-timestamp}.{raw body}`, signed with the endpoint's secret. Pass the secret to your verification library exactly as it appears in the UI; the library handles the `whsec_` prefix. Use the raw request body. Reserializing parsed JSON can change the bytes and invalidate the signature. **TypeScript** ```bash npm install standardwebhooks ``` ```ts import express from "express"; import { Webhook } from "standardwebhooks"; const wh = new Webhook(process.env.JUDGMENT_WEBHOOK_SECRET!); const app = express(); app.post( "/webhooks/judgment", express.raw({ type: "application/json" }), (req, res) => { let event; try { event = wh.verify(req.body, req.headers as Record); } catch { return res.sendStatus(400); } res.sendStatus(204); void handleEvent(event); }, ); ``` **Python** ```bash pip install standardwebhooks ``` ```python import os from fastapi import FastAPI, Request, Response from standardwebhooks.webhooks import Webhook, WebhookVerificationError wh = Webhook(os.environ["JUDGMENT_WEBHOOK_SECRET"]) app = FastAPI() @app.post("/webhooks/judgment") async def receive(request: Request) -> Response: try: event = wh.verify(await request.body(), dict(request.headers)) except WebhookVerificationError: return Response(status_code=400) enqueue(event) return Response(status_code=204) ``` **Go** ```bash go get github.com/standard-webhooks/standard-webhooks/libraries/go ``` ```go package main import ( "io" "net/http" "os" standardwebhooks "github.com/standard-webhooks/standard-webhooks/libraries/go" ) func main() { wh, err := standardwebhooks.NewWebhook(os.Getenv("JUDGMENT_WEBHOOK_SECRET")) if err != nil { panic(err) } http.HandleFunc("/webhooks/judgment", func(w http.ResponseWriter, r *http.Request) { body, err := io.ReadAll(r.Body) if err != nil { w.WriteHeader(http.StatusBadRequest) return } if err := wh.Verify(body, r.Header); err != nil { w.WriteHeader(http.StatusBadRequest) return } w.WriteHeader(http.StatusNoContent) go handleEvent(body) }) http.ListenAndServe(":8080", nil) } ``` Reference libraries are also available for Ruby, Rust, PHP, Java/Kotlin, C#, and Elixir. Treat a delivery as valid if any space-separated `v1` signature matches. After signature validation, check that `webhook-timestamp` is within five minutes of your system time to prevent replay attacks. Use a constant-time comparison if you implement verification directly. ### Delivery behavior Judgment aborts an attempt that takes longer than 10 seconds. Respond with a `2xx` status after validating the signature, then process the event asynchronously. Each delivery is attempted up to four times total: the initial attempt plus at most three retries. Retries use exponential backoff with jitter, starting at 500 ms. Timeouts, network errors, and `408`, `429`, or `5xx` responses are retried; other non-`2xx` responses are permanent failures. Redirects are not followed, so configure the final HTTPS URL directly. Delivery is at least once. The `webhook-id` remains stable across retries, so use it as the idempotency key. Frequency and cooldown limits apply before a webhook event is sent.