Evaluate traces

NOT YET IMPLEMENTED

POST/v1/projects/{projectId}/evaluate/traces

Authorization

AuthorizationPersonal API key · headerrequired

Your personal API key from Settings > Personal API Keys, sent as Bearer <key>.

X-Organization-IdOrganization ID · headerrequired

The organization to act in. Copy it with Copy organization ID in the organization menu or the settings sidebar.

Path parameters

projectIdstringrequired

Request body

requiredapplication/json

idstring | null

Show properties

Any of:

string

string

null

null

project_idstringrequired

eval_namestringrequired

modelstring | null

Show properties

Any of:

string

string

null

null

created_atstring | null

Show properties

Any of:

string

string

null

null

user_idstring

scorersany[] | null

Show properties

Any of:

any[]

Array of any

any

null

null

custom_scorersBaseScorer[]required

default: []

Show properties

Array of BaseScorer

score_typestringrequired

namestringrequired

class_namestring | null

Show properties

Any of:

string

string

null

null

scorenumber | null

Show properties

Any of:

number

number

null

null

minimum_score_rangenumber

default: 0

maximum_score_rangenumber

default: 1

score_breakdownobject | null

Show properties

Any of:

object

object

null

null

reasonstring | object

Show properties

Any of:

string

string

object

object

successboolean | null

Show properties

Any of:

boolean

boolean

null

null

modelstring | null

Show properties

Any of:

string

string

null

null

errorstring | null

Show properties

Any of:

string

string

null

null

additional_metadataobject | null

Show properties

Any of:

object

object

null

null

userstring | null

Show properties

Any of:

string

string

null

null

server_hostedboolean | null

Show properties

Any of:

boolean

boolean

null

null

using_native_modelboolean | null

Show properties

Any of:

boolean

boolean

null

null

required_paramsstring[] | null

Show properties

Any of:

string[]

Array of string

string

null

null

strict_modeboolean | null

Show properties

Any of:

boolean

boolean

null

null

judgment_scorersScorerConfig[]required

default: []

Show properties

Array of ScorerConfig

score_typestring | null

Show properties

Any of:

string

string

null

null

namestringrequired

modelstring | null

Show properties

Any of:

string

string

null

null

required_paramsstring[] | null

Show properties

Any of:

string[]

Array of string

string

null

null

kwargsobject | null

Show properties

Any of:

object

object

null

null

result_typestring

Allowed:numericbinarycategorical

trace_and_span_idsany[][]required

min items 1

is_offlinebooleanrequired

default: false

is_behaviorbooleanrequired

default: false

Responses

501

Response for status 501

errorstringrequired

messagestring

codestring

detailsstring[]

Try it

Server

Base URLCustom base URLOverrides the selected server when set.

Authorization

Personal API keyOrganization IDRemember on this deviceInclude my values in samples

Parameters

projectIdpath*

Bodyapplication/json

Request body

Request

curl -X POST 'https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/evaluate/traces' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'X-Organization-Id: YOUR_ORGANIZATION_ID' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "id": "string",
  "project_id": "string",
  "eval_name": "string",
  "model": "string",
  "created_at": "string",
  "user_id": "string",
  "scorers": [
    null
  ],
  "custom_scorers": [],
  "judgment_scorers": [],
  "trace_and_span_ids": [
    [
      "string",
      "string"
    ]
  ],
  "is_offline": false,
  "is_behavior": false
}'

const response = await fetch("https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/evaluate/traces", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "X-Organization-Id": "YOUR_ORGANIZATION_ID",
    "Accept": "application/json",
    "Content-Type": "application/json"
  },
  body: "{\n  \"id\": \"string\",\n  \"project_id\": \"string\",\n  \"eval_name\": \"string\",\n  \"model\": \"string\",\n  \"created_at\": \"string\",\n  \"user_id\": \"string\",\n  \"scorers\": [\n    null\n  ],\n  \"custom_scorers\": [],\n  \"judgment_scorers\": [],\n  \"trace_and_span_ids\": [\n    [\n      \"string\",\n      \"string\"\n    ]\n  ],\n  \"is_offline\": false,\n  \"is_behavior\": false\n}"
});

import requests

response = requests.post(
    "https://api.judgmentlabs.ai/v1/projects/YOUR_PROJECT_ID/evaluate/traces",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "X-Organization-Id": "YOUR_ORGANIZATION_ID",
        "Accept": "application/json",
        "Content-Type": "application/json"
    },
    data="{\n  \"id\": \"string\",\n  \"project_id\": \"string\",\n  \"eval_name\": \"string\",\n  \"model\": \"string\",\n  \"created_at\": \"string\",\n  \"user_id\": \"string\",\n  \"scorers\": [\n    null\n  ],\n  \"custom_scorers\": [],\n  \"judgment_scorers\": [],\n  \"trace_and_span_ids\": [\n    [\n      \"string\",\n      \"string\"\n    ]\n  ],\n  \"is_offline\": false,\n  \"is_behavior\": false\n}",
)

Response

{
  "error": "string",
  "message": "string",
  "code": "string",
  "details": [
    "string"
  ]
}