Add cases to a test suite's draft

Add cases to the test suite’s draft. Each case has an input for the agent and an expected object keyed by judge id: {judge_major_version, judge_minor_version, op, expected}. A check passes when the judge’s output <op> expected; op is eq, ne, lt, lte, gt, gte, in or not_in. Publish the draft to run it.

POST/v1/projects/{projectId}/test-suites/{testSuiteId}/cases

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstring<uuid>required

testSuiteIdstring<uuid>required

Request body

requiredapplication/json

casesTestCaseInput[]required

min items 1 · max items 1000

Show properties

Array of TestCaseInput

namestring | null

Show properties

Any of:

string

string

null

null

inputanyrequired

What the agent gets.

expectedobject

tagsstring[]

metadataobject

Responses

200

Response for status 200

case_idsstring[]required

404

Response for status 404

errorstringrequired

messagestring

codestring

detailsstring[]

409

Response for status 409

errorstringrequired

messagestring

codestring

detailsstring[]

422

Response for status 422

errorstringrequired

messagestring

codestring

detailsstring[]

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*testSuiteIdpath*

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/cases' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "cases": [
    {
      "name": "string",
      "input": null,
      "expected": {},
      "tags": [
        "string"
      ],
      "metadata": {}
    }
  ]
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/cases", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"cases\": [\n    {\n      \"name\": \"string\",\n      \"input\": null,\n      \"expected\": {},\n      \"tags\": [\n        \"string\"\n      ],\n      \"metadata\": {}\n    }\n  ]\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/YOUR_TEST_SUITE_ID/cases",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"cases\": [\n    {\n      \"name\": \"string\",\n      \"input\": null,\n      \"expected\": {},\n      \"tags\": [\n        \"string\"\n      ],\n      \"metadata\": {}\n    }\n  ]\n}",
)

Response

{
  "case_ids": [
    "string"
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}