Configure automations and alerts

Run actions when a completed trace matches behavior or trace conditions.

Automations evaluate conditions against each completed trace. When a trace matches, Judgment can notify a team, add the trace to a dataset, or run a behavior evaluation. This guide creates one automation and verifies its action.

Prerequisites

• An instrumented agent sending completed traces to the target project
• A known condition you can trigger safely in a test trace
• Access to the notification destination or dataset used by the action

For a behavior condition, first create and monitor the behavior. For a trace condition, confirm that the required duration, cost, exception, or span attribute is present on incoming traces.

1. Create the automation

Open Monitoring, select the Automations tab, and then select New Automation. Enter a name that states the event and response, such as Unsafe response to Slack. Add a description when the threshold or owner would not otherwise be clear.

Leave the automation enabled so you can verify it with new traffic.

2. Add conditions

Under Trigger on, select Add trigger and choose one of the supported condition families:

For example, choose Behaviors > Behavior triggered, then select the unsafe behavior beneath its owning judge.

Add more conditions when necessary. Choose AND conditions when every condition must match the same trace, or OR conditions when any one is sufficient. Keep the first verification to one unambiguous condition.

3. Choose at least one action

Under Actions, select Add action. An automation can run more than one action:

Choose an action you can observe directly. For a first test, adding to a small dataset or sending to a test notification destination makes verification straightforward.

4. Set frequency and cooldown limits

Run actions only if conditions matched at least controls action frequency. Its default is 1 time in the last 1 second, so each matching trace can run the actions.

Once actions are triggered, wait controls the cooldown. Its default is 0 seconds, so there is no delay before the next eligible action.

Use frequency to require several matching traces within a rolling window. Use cooldown to prevent repeated notifications or actions after the threshold is met. Both support seconds, minutes, hours, and days.

5. Save and verify

Save the automation, then send one controlled trace that satisfies its condition. Wait for the root span to complete and verify both sides:

• In Logs > Traces, the trace appears with the automation in its Automations column.
• The configured action happened: the message arrived, the trace appeared in the dataset, or the selected behavior produced a result.

If the condition matched but the action did not run, check whether the frequency threshold has been reached and whether the automation is still in its cooldown period. Also confirm that the automation is enabled and that the trace completed after you saved it.

Condition and action boundaries

• Conditions and actions operate on the same finalized trace.
• A behavior condition uses a binary or categorical signal defined under its owning judge. Numeric judge outputs are not behaviors.
• Run behavior can evaluate selected behaviors after a trace first matches a cheaper trace condition, such as a span attribute.
• Add to dataset preserves matching traces for later review and offline testing.
• Email, Slack, PagerDuty, and webhook actions are subject to the same frequency and cooldown limits.

For the complete behavior model, see Judges, results, and behaviors. For trace condition semantics, see Trace conventions.

Webhook reference

Webhook setup and payload details are reference material for teams that choose the webhook action. They do not change the automation setup path above.

Configure the endpoint

Judgment implements the Standard Webhooks specification, so you can verify deliveries with any conforming library.

Request format

Every delivery is a POST with a JSON body and these headers:

Event payload

Version 1 defines one event type, automation.rule.triggered:

{
  "id": "8f2b7ac1-4d1e-4c3a-9f70-6f6e1b2c9a05",
  "type": "automation.rule.triggered",
  "version": "1",
  "test": false,
  "created_at": "2026-07-20T18:42:11.204Z",
  "organization_id": "0b0f3a2a-3a6c-4a4d-9c6a-2f1f0a6d5e11",
  "project_id": "3c9d1f77-2a44-4f2e-bb84-0f2c9c3a7d10",
  "data": {
    "automation": {
      "id": "b6d4d1e2-9f3b-4a71-8b1f-2c0f7a5e4c33",
      "name": "Slow checkout agent"
    },
    "trace": {
      "id": "9a7c1f0e2b3d4c5a6e7f8091a2b3c4d5",
      "started_at": "2026-07-20T18:42:09.980Z",
      "duration_ms": 1250,
      "llm_cost_usd": 0.0025,
      "has_error": false
    },
    "condition_evaluations": [
      {
        "metric": "duration",
        "unit": "seconds",
        "comparison": "gt",
        "threshold": 1,
        "observed_value": 1.25,
        "matched": true
      }
    ]
  }
}

Each entry in condition_evaluations describes one condition:

Conditions skipped during evaluation are omitted. Raw trace content, span attributes, and error messages are never included in the payload. Use data.trace.id to fetch those from the API. A breaking payload change ships as a new version.

For a categorical behavior condition, metric does not contain the behavior value. It contains the owning judge’s judge_name; threshold contains the selected behavior value, and observed_value contains the judge’s returned choice.

Verify the signature

The webhook-signature header contains a base64-encoded HMAC-SHA256 signature of {webhook-id}.{webhook-timestamp}.{raw body}, signed with the endpoint’s secret. Pass the secret to your verification library exactly as it appears in the UI; the library handles the whsec_ prefix.

Use the raw request body. Reserializing parsed JSON can change the bytes and invalidate the signature.

npm install standardwebhooks

import express from "express";
import { Webhook } from "standardwebhooks";

const wh = new Webhook(process.env.JUDGMENT_WEBHOOK_SECRET!);
const app = express();

app.post(
  "/webhooks/judgment",
  express.raw({ type: "application/json" }),
  (req, res) => {
    let event;
    try {
      event = wh.verify(req.body, req.headers as Record<string, string>);
    } catch {
      return res.sendStatus(400);
    }

    res.sendStatus(204);
    void handleEvent(event);
  },
);

pip install standardwebhooks

import os

from fastapi import FastAPI, Request, Response
from standardwebhooks.webhooks import Webhook, WebhookVerificationError

wh = Webhook(os.environ["JUDGMENT_WEBHOOK_SECRET"])
app = FastAPI()


@app.post("/webhooks/judgment")
async def receive(request: Request) -> Response:
    try:
        event = wh.verify(await request.body(), dict(request.headers))
    except WebhookVerificationError:
        return Response(status_code=400)

    enqueue(event)
    return Response(status_code=204)

go get github.com/standard-webhooks/standard-webhooks/libraries/go

package main

import (
    "io"
    "net/http"
    "os"

    standardwebhooks "github.com/standard-webhooks/standard-webhooks/libraries/go"
)

func main() {
    wh, err := standardwebhooks.NewWebhook(os.Getenv("JUDGMENT_WEBHOOK_SECRET"))
    if err != nil {
        panic(err)
    }

    http.HandleFunc("/webhooks/judgment", func(w http.ResponseWriter, r *http.Request) {
        body, err := io.ReadAll(r.Body)
        if err != nil {
            w.WriteHeader(http.StatusBadRequest)
            return
        }
        if err := wh.Verify(body, r.Header); err != nil {
            w.WriteHeader(http.StatusBadRequest)
            return
        }

        w.WriteHeader(http.StatusNoContent)
        go handleEvent(body)
    })

    http.ListenAndServe(":8080", nil)
}

Reference libraries are also available for Ruby, Rust, PHP, Java/Kotlin, C#, and Elixir.

Treat a delivery as valid if any space-separated v1 signature matches. After signature validation, check that webhook-timestamp is within five minutes of your system time to prevent replay attacks. Use a constant-time comparison if you implement verification directly.

Delivery behavior

Judgment aborts an attempt that takes longer than 10 seconds. Respond with a 2xx status after validating the signature, then process the event asynchronously.

Each delivery is attempted up to four times total: the initial attempt plus at most three retries. Retries use exponential backoff with jitter, starting at 500 ms. Timeouts, network errors, and 408, 429, or 5xx responses are retried; other non-2xx responses are permanent failures. Redirects are not followed, so configure the final HTTPS URL directly.

Delivery is at least once. The webhook-id remains stable across retries, so use it as the idempotency key. Frequency and cooldown limits apply before a webhook event is sent.