Get test run results

Each case of the run with its output trace, whether it is scored, its outcome and check results; filter to failed cases or one judge, a page at a time.

GET/v1/projects/{projectId}/test-suites/runs/{runId}/results

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstring<uuid>required

runIdstring<uuid>required

Query parameters

failed_onlyboolean

Only failed cases; with judge_id, only cases where that judge's check failed.

judge_idstring

Only this judge's checks, and only cases that have a check by it.

offsetinteger

min 0

limitinteger

Cases per page; omit for every case.

min 1 · max 100

Responses

200

Response for status 200

resultsobject[]required

Show properties

Array of object

case_idstring<uuid>required

trace_idstring | nullrequired

Show properties

Any of:

string

string

null

null

scored_atstring<date-time> | nullrequired

Show properties

Any of:

string<date-time>

string<date-time>

null

null

passedboolean | nullrequired

Show properties

Any of:

boolean

boolean

null

null

checksobject[]required

Show properties

Array of object

judge_idstring<uuid>required

The check's key in the case's expected.

judge_namestring | nullrequired

Show properties

Any of:

string

string

null

null

judge_major_versionnumberrequired

judge_minor_versionnumberrequired

opstringrequired

default: "eq"

Allowed:eqneltltegtgteinnot_in

expectedanyrequired

The golden value.

judge_outputanyrequired

What the judge returned; NULL when it errored.

passedbooleanrequired

reasonstring | nullrequired

Show properties

Any of:

string

string

null

null

error_messagestring | nullrequired

Show properties

Any of:

string

string

null

null

pending_checksobject[]required

Checks in the case's expected that have no result yet.

Show properties

Array of object

judge_idstring<uuid>required

judge_namestring | nullrequired

Show properties

Any of:

string

string

null

null

judge_major_versionnumberrequired

judge_minor_versionnumberrequired

statusstringrequired

The judge run's status; not_queued until the case's trace is scored.

default: "not_queued"

Allowed:not_queuedpendingrunningcompletederrorcancelled

queued_atstring<date-time> | nullrequired

Show properties

Any of:

string<date-time>

string<date-time>

null

null

started_atstring<date-time> | nullrequired

Show properties

Any of:

string<date-time>

string<date-time>

null

null

error_messagestring | nullrequired

Show properties

Any of:

string

string

null

null

next_offsetinteger | nullrequired

Show properties

Any of:

integer

integer

null

null

404

Response for status 404

errorstringrequired

messagestring

codestring

detailsstring[]

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*runIdpath*failed_onlyqueryOnly failed cases; with judge_id, only cases where that judge's check failed.judge_idqueryOnly this judge's checks, and only cases that have a check by it.offsetquerylimitqueryCases per page; omit for every case.

Request

curl -X GET 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/runs/YOUR_RUN_ID/results' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/runs/YOUR_RUN_ID/results", {
  method: "GET",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json"
  }
});

import requests

response = requests.get(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/test-suites/runs/YOUR_RUN_ID/results",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json"
    },
)

Response

{
  "results": [
    {
      "case_id": "c74269e5-1f97-4e20-9164-ffbe3494d8d6",
      "trace_id": "string",
      "scored_at": "2019-08-24T14:15:22Z",
      "passed": true,
      "checks": [
        {
          "judge_id": "968a8026-4d88-467b-937b-009ebfad94d9",
          "judge_name": "string",
          "judge_major_version": 0,
          "judge_minor_version": 0,
          "op": "eq",
          "expected": null,
          "judge_output": null,
          "passed": true,
          "reason": "string",
          "error_message": "string"
        }
      ],
      "pending_checks": [
        {
          "judge_id": "968a8026-4d88-467b-937b-009ebfad94d9",
          "judge_name": "string",
          "judge_major_version": 0,
          "judge_minor_version": 0,
          "status": "not_queued",
          "queued_at": "2019-08-24T14:15:22Z",
          "started_at": "2019-08-24T14:15:22Z",
          "error_message": "string"
        }
      ]
    }
  ],
  "next_offset": 0
}

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}