Evaluate recent traces

Request evaluation of the most recent traces across all time (up to limit of 1000, default 100) with current production judges, optionally restricted to named judges. Evaluation is asynchronous and costs JUs. Each judge type is checked against its own usage limit; the response lists admitted and skipped judges per trace and any unknown judge names.

POST/v1/projects/{projectId}/traces/evaluate-all

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstringrequired

Request body

requiredapplication/json

judge_namesstring[]

Restrict evaluation to these judge names

limitinteger

Max number of traces to evaluate, taken from the most recent across all time

min 1 · max 1000 · default: 100

Responses

200

Response for status 200

enqueuedbooleanrequired

True when at least one judge was queued for at least one trace

limitnumberrequired

tracesobject[]required

Show properties

Array of object

trace_idstringrequired

admittedstring[]required

Judge names queued for this trace

skippedobject[]required

Judges not queued for this trace, each with the reason

Show properties

Array of object

judgestringrequired

reasonstringrequired

on_demand_not_requested: on-demand judge, name it in judge_names to run it. sampled_out: excluded by the sampling rate. trigger_not_matched: the trace does not meet the run conditions, or the environment filter when the judge has both. environment_not_matched: the trace environment is not in the environment filter of a judge with no run conditions. no_session: session judge on a trace with no session. session_already_claimed: this session was already evaluated by the judge. usage_limited: the judge type is over the organization usage limit. rate_limited: the evaluation rate limit was reached.

Allowed:on_demand_not_requestedsampled_outtrigger_not_matchedenvironment_not_matchedno_sessionsession_already_claimedusage_limitedrate_limited

unknown_judge_namesstring[]required

Requested judge names that match no production judge in this project

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate-all' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "judge_names": [
    "string"
  ],
  "limit": 100
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate-all", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"judge_names\": [\n    \"string\"\n  ],\n  \"limit\": 100\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate-all",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"judge_names\": [\n    \"string\"\n  ],\n  \"limit\": 100\n}",
)

Response

{
  "enqueued": true,
  "limit": 0,
  "traces": [
    {
      "trace_id": "string",
      "admitted": [
        "string"
      ],
      "skipped": [
        {
          "judge": "string",
          "reason": "on_demand_not_requested"
        }
      ]
    }
  ],
  "unknown_judge_names": [
    "string"
  ]
}