Evaluate stored traces

Request evaluation of specific stored traces with current production judges, optionally restricted to named judges. Evaluation is asynchronous and costs JUs. Each judge type is checked against its own usage limit; the response lists admitted and skipped judges per trace and any unknown trace IDs or judge names.

POST/v1/projects/{projectId}/traces/evaluate

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstringrequired

Request body

requiredapplication/json

trace_idsstring[]required

Trace IDs to evaluate

min items 1 · max items 100

judge_namesstring[]

Restrict evaluation to these judge names

Responses

200

Response for status 200

enqueuedbooleanrequired

True when at least one judge was queued for at least one trace

trace_countnumberrequired

Number of trace IDs in the request

tracesobject[]required

Show properties

Array of object

trace_idstringrequired

admittedstring[]required

Judge names queued for this trace

skippedobject[]required

Judges not queued for this trace, each with the reason

Show properties

Array of object

judgestringrequired

reasonstringrequired

on_demand_not_requested: on-demand judge, name it in judge_names to run it. sampled_out: excluded by the sampling rate. trigger_not_matched: the trace does not meet the run conditions, or the environment filter when the judge has both. environment_not_matched: the trace environment is not in the environment filter of a judge with no run conditions. no_session: session judge on a trace with no session. session_already_claimed: this session was already evaluated by the judge. usage_limited: the judge type is over the organization usage limit. rate_limited: the evaluation rate limit was reached.

Allowed:on_demand_not_requestedsampled_outtrigger_not_matchedenvironment_not_matchedno_sessionsession_already_claimedusage_limitedrate_limited

unknown_trace_idsstring[]required

Requested trace IDs with no stored root span in this project; they were not evaluated

unknown_judge_namesstring[]required

Requested judge names that match no production judge in this project

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "trace_ids": [
    "string"
  ],
  "judge_names": [
    "string"
  ]
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"trace_ids\": [\n    \"string\"\n  ],\n  \"judge_names\": [\n    \"string\"\n  ]\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/traces/evaluate",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"trace_ids\": [\n    \"string\"\n  ],\n  \"judge_names\": [\n    \"string\"\n  ]\n}",
)

Response

{
  "enqueued": true,
  "trace_count": 0,
  "traces": [
    {
      "trace_id": "string",
      "admitted": [
        "string"
      ],
      "skipped": [
        {
          "judge": "string",
          "reason": "on_demand_not_requested"
        }
      ]
    }
  ],
  "unknown_trace_ids": [
    "string"
  ],
  "unknown_judge_names": [
    "string"
  ]
}