Start a test run

Runs the test suite’s latest published version over the cases the filter selects; each case is scored by the checks in its expected. Run your agent once per returned case on your normal tracer, with judgment.test.run_id and judgment.test.case_id set; traces are attached and scored as they arrive.

POST/v1/projects/{projectId}/test-suites/{testSuiteId}/runs

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstring<uuid>required

testSuiteIdstring<uuid>required

Request body

requiredapplication/json

case_filterTestCaseFilter

Show properties

Any of:

object

kindstringrequired

object

kindstringrequired

tagsstring[]required

min items 1

object

kindstringrequired

case_idsstring<uuid>[]required

min items 1

object

kindstringrequired

run_idstring<uuid>required

candidatestring

What is being tested, for example a git SHA or agent version.

candidate_metaobject

triggerstring

default: "api"

Allowed:manualcischeduleapi

Responses

200

Response for status 200

run_idstring<uuid>required

test_suite_versionintegerrequired

casesobject[]required

Run the agent once per case, with judgment.test.run_id and judgment.test.case_id set on its trace.

Show properties

Array of object

case_idstring<uuid>required

inputanyrequired

What the agent gets.

headersobjectrequired

Send these with the request to a deployed agent; Tracer.continue_trace turns them into the test attributes on its trace.

404

Response for status 404

errorstringrequired

messagestring

codestring

detailsstring[]

409

Response for status 409

errorstringrequired

messagestring

codestring

detailsstring[]

422

Response for status 422

errorstringrequired

messagestring

codestring

detailsstring[]

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*testSuiteIdpath*

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/runs' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "case_filter": {
    "kind": "all"
  },
  "candidate": "string",
  "candidate_meta": {},
  "trigger": "manual"
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/runs", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"case_filter\": {\n    \"kind\": \"all\"\n  },\n  \"candidate\": \"string\",\n  \"candidate_meta\": {},\n  \"trigger\": \"manual\"\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/runs",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"case_filter\": {\n    \"kind\": \"all\"\n  },\n  \"candidate\": \"string\",\n  \"candidate_meta\": {},\n  \"trigger\": \"manual\"\n}",
)

Response

{
  "run_id": "dded282c-8ebd-44cf-8ba5-9a234973d1ec",
  "test_suite_version": 0,
  "cases": [
    {
      "case_id": "c74269e5-1f97-4e20-9164-ffbe3494d8d6",
      "input": null,
      "headers": {}
    }
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}