Test optimistic eval candidate

Backtest unsaved preprocessing source against one historical trace with an exact saved judge version. Runs the candidate (preprocessing + JEV) and an ordinary agent judge in parallel and returns both outcomes and whether they match. Consumes evaluation usage; saves no scores.

POST/v1/projects/{projectId}/artifacts/optimistic-eval-tests

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstringrequired

Request body

requiredapplication/json

trace_idstringrequired

The historical trace to extract evidence from

min length 1 · max length 128

source_codestringrequired

A complete JavaScript function expression: async ({ traceId, sql }) => { ... }. Return JSON evidence; throw when validation fails.

max length 100000

judge_idstring<uuid>required

The judge in this project

matches ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$

judge_version_idstring<uuid>required

The exact judge version ID returned by get_judge

matches ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$

Responses

200

Response for status 200

judge_idstring<uuid>required

matches ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$

judge_version_idstring<uuid>required

matches ^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|ffffffff-ffff-ffff-ffff-ffffffffffff)$

judge_major_versionintegerrequired

min 0 · max 9007199254740991

judge_minor_versionintegerrequired

min 0 · max 9007199254740991

score_typestringrequired

Allowed:binarycategorical

trace_idstringrequired

resultobjectrequired

Show properties

One of:

object

statusstringrequired

verdictboolean | stringrequired

reasonnullrequired

errornullrequired

confidencenumberrequired

min 0 · max 1

duration_msnumberrequired

min 0

artifact_duration_msnumberrequired

min 0

evidence_size_bytesintegerrequired

min 0 · max 9007199254740991

object

statusstringrequired

verdictnullrequired

reasonstringrequired

Allowed:artifact_failedevidence_and_rubric_too_largemodel_failedlow_confidencetimeout

errornullrequired

confidencenumber | nullrequired

Show properties

Any of:

number

number

null

null

duration_msnumberrequired

min 0

artifact_duration_msnumber | nullrequired

Show properties

Any of:

number

number

null

null

evidence_size_bytesinteger | nullrequired

Show properties

Any of:

integer

integer

null

null

object

statusstringrequired

verdictnullrequired

reasonnullrequired

errorstringrequired

confidencenullrequired

duration_msnumberrequired

min 0

artifact_duration_msnullrequired

evidence_size_bytesnullrequired

ordinary_agentobjectrequired

Show properties

One of:

object

statusstringrequired

verdictboolean | stringrequired

reasonstringrequired

errornullrequired

duration_msnumberrequired

min 0

object

statusstringrequired

verdictnullrequired

reasonnullrequired

errorstringrequired

duration_msnumberrequired

min 0

matchesboolean | nullrequired

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*

Bodyapplication/json

trace_idstring*The historical trace to extract evidence fromsource_codestring*A complete JavaScript function expression: async ({ traceId, sql }) => { ... }. Return JSON evidence; throw when validation fails.judge_idstring*The judge in this projectjudge_version_idstring*The exact judge version ID returned by get_judge

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/artifacts/optimistic-eval-tests' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "trace_id": "string",
  "source_code": "string",
  "judge_id": "968a8026-4d88-467b-937b-009ebfad94d9",
  "judge_version_id": "85fd0392-86f2-40e7-9e56-5ca06ee20f82"
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/artifacts/optimistic-eval-tests", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"trace_id\": \"string\",\n  \"source_code\": \"string\",\n  \"judge_id\": \"968a8026-4d88-467b-937b-009ebfad94d9\",\n  \"judge_version_id\": \"85fd0392-86f2-40e7-9e56-5ca06ee20f82\"\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/artifacts/optimistic-eval-tests",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"trace_id\": \"string\",\n  \"source_code\": \"string\",\n  \"judge_id\": \"968a8026-4d88-467b-937b-009ebfad94d9\",\n  \"judge_version_id\": \"85fd0392-86f2-40e7-9e56-5ca06ee20f82\"\n}",
)

Response

{
  "judge_id": "968a8026-4d88-467b-937b-009ebfad94d9",
  "judge_version_id": "85fd0392-86f2-40e7-9e56-5ca06ee20f82",
  "judge_major_version": 0,
  "judge_minor_version": 0,
  "score_type": "binary",
  "trace_id": "string",
  "result": {
    "status": "accepted",
    "verdict": true,
    "reason": null,
    "error": null,
    "confidence": 0,
    "duration_ms": 0,
    "artifact_duration_ms": 0,
    "evidence_size_bytes": 0
  },
  "ordinary_agent": {
    "status": "completed",
    "verdict": true,
    "reason": "string",
    "error": null,
    "duration_ms": 0
  },
  "matches": true
}