Judgeval API

V1 and OTEL endpoints.

Version 1.0.0

Base URLhttps://api.judgmentlabs.ai

OTEL

POST
/otel/v1/traces

V1

GET
List organizations available to the API key/v1/organizations
GET
List organization memberships/v1/organizations/memberships
GET
List projects in the selected organization/v1/projects
POST
Add new project/v1/projects
DELETE
Delete project/v1/projects/{projectId}
GET
/v1/projects/{projectId}/agent-versions
POST
/v1/projects/{projectId}/agent-versions
GET
/v1/projects/{projectId}/agent-versions/{versionId}
PUT
/v1/projects/{projectId}/agent-versions/{versionId}/tags
GET
Get artifact revision/v1/projects/{projectId}/artifact-revisions
POST
Create artifact/v1/projects/{projectId}/artifacts
POST
Create artifact revision/v1/projects/{projectId}/artifacts/{artifactId}/revisions
POST
Test optimistic eval candidate/v1/projects/{projectId}/artifacts/optimistic-eval-tests
POST
Test preprocessing candidate/v1/projects/{projectId}/artifacts/preprocessing-tests
GET
List attribute mappers/v1/projects/{projectId}/attribute-mappers
POST
Create attribute mapper/v1/projects/{projectId}/attribute-mappers
GET
Get attribute mapper/v1/projects/{projectId}/attribute-mappers/{configId}
PATCH
Update attribute mapper/v1/projects/{projectId}/attribute-mappers/{configId}
DELETE
Delete attribute mapper/v1/projects/{projectId}/attribute-mappers/{configId}
PATCH
Update judge label/v1/projects/{projectId}/behaviors/{behaviorId}
GET
Get judge label/v1/projects/{projectId}/behaviors/{behaviorId}/analytics
GET
Get conversation display rules/v1/projects/{projectId}/conversation-display-rules
PUT
Replace conversation display rules/v1/projects/{projectId}/conversation-display-rules
POST
Create offline-test dataset/v1/projects/{projectId}/datasets
GET
Fetch offline-test datasets/v1/projects/{projectId}/datasets
DELETE
Delete offline-test dataset/v1/projects/{projectId}/datasets/{datasetIdentifier}
GET
Fetch offline-test dataset/v1/projects/{projectId}/datasets/{datasetIdentifier}
POST
Insert examples to offline-test dataset/v1/projects/{projectId}/datasets/{datasetIdentifier}/examples
POST
Remove examples from offline-test dataset/v1/projects/{projectId}/datasets/{datasetIdentifier}/examples/remove
GET
Fetch dataset page/v1/projects/{projectId}/datasets/{datasetIdentifier}/page
GET
Fetch dataset versions/v1/projects/{projectId}/datasets/{datasetIdentifier}/versions
POST
Queue example evaluation/v1/projects/{projectId}/eval-queue/examples
POST
Queue trace evaluation/v1/projects/{projectId}/eval-queue/traces
POST
Log evaluation results/v1/projects/{projectId}/eval-results
POST
Log evaluation results/v1/projects/{projectId}/eval-results/examples
POST
Evaluate examples/v1/projects/{projectId}/evaluate/examples
POST
Evaluate traces/v1/projects/{projectId}/evaluate/traces
GET
Fetch offline-test run via experiment alias/v1/projects/{projectId}/experiments/{runId}
GET
List issues/v1/projects/{projectId}/issues
POST
Create issue/v1/projects/{projectId}/issues
GET
Get issue/v1/projects/{projectId}/issues/{issueId}
PATCH
Update issue/v1/projects/{projectId}/issues/{issueId}
GET
List judges/v1/projects/{projectId}/judges
POST
Create judge/v1/projects/{projectId}/judges
PATCH
Update judge/v1/projects/{projectId}/judges/{judgeId}
DELETE
Delete judge/v1/projects/{projectId}/judges/{judgeId}
POST
Set judge action groups/v1/projects/{projectId}/judges/{judgeId}/action-groups
POST
Get judge/v1/projects/{projectId}/judges/{judgeId}/lookup
PUT
Set judge version optimistic eval/v1/projects/{projectId}/judges/{judgeId}/versions/{judgeVersionId}/optimistic-eval
PUT
Set judge version preprocessing artifact/v1/projects/{projectId}/judges/{judgeId}/versions/{judgeVersionId}/preprocessor-artifact-revision
GET
List judge action groups/v1/projects/{projectId}/judges/action-groups
GET
List memory entries/v1/projects/{projectId}/memory/entries
PUT
Write a memory file/v1/projects/{projectId}/memory/files
POST
Fetch memory files/v1/projects/{projectId}/memory/files/fetch
POST
Search memory files/v1/projects/{projectId}/memory/search
POST
Insert prompt/v1/projects/{projectId}/prompts
GET
Fetch prompt/v1/projects/{projectId}/prompts/{name}
POST
Tag prompt/v1/projects/{projectId}/prompts/{name}/tags
DELETE
Untag prompt/v1/projects/{projectId}/prompts/{name}/tags
GET
Get prompt versions/v1/projects/{projectId}/prompts/{name}/versions
GET
List scheduled tasks/v1/projects/{projectId}/scheduled-tasks
POST
Create a scheduled task/v1/projects/{projectId}/scheduled-tasks
GET
Get a scheduled task/v1/projects/{projectId}/scheduled-tasks/{taskId}
PATCH
Update a scheduled task/v1/projects/{projectId}/scheduled-tasks/{taskId}
GET
Fetch scorers/v1/projects/{projectId}/scorers
GET
Check if scorer exists/v1/projects/{projectId}/scorers/{name}/exists
POST
Upload custom scorer (deprecated)/v1/projects/{projectId}/scorers/custom
GET
Check if custom scorer exists/v1/projects/{projectId}/scorers/custom/{name}/exists
POST
Upload custom scorer bundle/v1/projects/{projectId}/scorers/custom/bundle
POST
Run a tenant-scoped virtual SQL query/v1/projects/{projectId}/sql
GET
Fetch test configs/v1/projects/{projectId}/test-configs
POST
Create test config/v1/projects/{projectId}/test-configs
GET
Fetch test config/v1/projects/{projectId}/test-configs/{testConfigId}
PATCH
Update test config/v1/projects/{projectId}/test-configs/{testConfigId}
DELETE
Delete test config/v1/projects/{projectId}/test-configs/{testConfigId}
GET
Fetch test runs/v1/projects/{projectId}/test-runs
POST
Prepare and queue test run/v1/projects/{projectId}/test-runs
GET
Fetch test run/v1/projects/{projectId}/test-runs/{testRunId}
POST
Cancel test run/v1/projects/{projectId}/test-runs/{testRunId}/cancel
GET
Fetch test run graph/v1/projects/{projectId}/test-runs/{testRunId}/graph
GET
Fetch test run items/v1/projects/{projectId}/test-runs/{testRunId}/items
GET
Fetch test run results/v1/projects/{projectId}/test-runs/{testRunId}/results
PATCH
Update test run status/v1/projects/{projectId}/test-runs/{testRunId}/status
PATCH
Apply test run pass-condition successes/v1/projects/{projectId}/test-runs/{testRunId}/success
POST
Create a test suite/v1/projects/{projectId}/test-suites
GET
List test suites/v1/projects/{projectId}/test-suites
DELETE
Delete a test suite with its runs/v1/projects/{projectId}/test-suites/{testSuiteId}
GET
List test cases in a version or the draft/v1/projects/{projectId}/test-suites/{testSuiteId}/cases
POST
Add cases to a test suite's draft/v1/projects/{projectId}/test-suites/{testSuiteId}/cases
PUT
Replace a case in the draft/v1/projects/{projectId}/test-suites/{testSuiteId}/cases/{caseId}
DELETE
Remove a case from the draft/v1/projects/{projectId}/test-suites/{testSuiteId}/cases/{caseId}
POST
Throw away the draft's changes/v1/projects/{projectId}/test-suites/{testSuiteId}/discard
POST
Copy the published cases into the draft to edit/v1/projects/{projectId}/test-suites/{testSuiteId}/draft
POST
Publish the draft as the next version/v1/projects/{projectId}/test-suites/{testSuiteId}/publish
POST
Start a test run/v1/projects/{projectId}/test-suites/{testSuiteId}/runs
GET
List test runs/v1/projects/{projectId}/test-suites/{testSuiteId}/runs
GET
Get a test run/v1/projects/{projectId}/test-suites/runs/{runId}
GET
Get test run results/v1/projects/{projectId}/test-suites/runs/{runId}/results
POST
Score a test run/v1/projects/{projectId}/test-suites/runs/{runId}/score
GET
List traces failing a trace health check/v1/projects/{projectId}/trace-health/checks/{checkId}/traces
GET
Get trace health summary/v1/projects/{projectId}/trace-health/summary
GET
Get trace health for one trace/v1/projects/{projectId}/trace-health/traces/{traceId}
POST
Add trace tags/v1/projects/{projectId}/traces/{traceId}/tags
POST
Evaluate stored traces/v1/projects/{projectId}/traces/evaluate
POST
Evaluate recent traces/v1/projects/{projectId}/traces/evaluate-all
POST
Resolve project by name/v1/projects/resolve/
GET
List project summaries/v1/projects/summaries
GET
Discover the virtual SQL schema/v1/sql/schema