OfflineTestsFactory

Create test configs and execute offline test runs. Access via `client.offlineTests`. A *test config* pairs a dataset with a set of platform judges; a *test run* evaluates one dataset version and stores per-example results.

const config = await client.offlineTests.createConfig("nightly", "golden-set", [
  "helpfulness",
  "faithfulness",
]);
const result = await client.offlineTests.run("nightly");
console.log(result?.uiResultsUrl);

createConfig()

Create a test config binding a dataset to a set of judges.

async function createConfig(name: string, dataset: string, judges: JudgeRef[], description?: string | undefined): Promise<TestConfig | null>

Parameters

PropType

namestring

Name for the test config.

Typestring

datasetstring

Dataset name or dataset id.

Typestring

judgesJudgeRef[]

Judges to attach (judge names, or { judgeId } / { name }).

TypeJudgeRef[]

description?string | undefined

Optional human-readable description.

Typestring | undefined

Returns

Promise<TestConfig | null>

getConfig()

Fetch a test config by id (UUID) or name.

async function getConfig(testConfig: string): Promise<TestConfig | null>

Parameters

PropType

testConfigstring

Typestring

Returns

Promise<TestConfig | null>

listConfigs()

List test configs in the project, optionally filtered to one dataset.

async function listConfigs(datasetId?: string | undefined): Promise<TestConfig[] | null>

Parameters

PropType

datasetId?string | undefined

Typestring | undefined

Returns

Promise<TestConfig[] | null>

deleteConfig()

Delete a test config by id. Returns false if the project is unresolved.

async function deleteConfig(testConfigId: string): Promise<boolean>

Parameters

PropType

testConfigIdstring

Typestring

Returns

Promise<boolean>

run()

Run an offline test for a test config.

Without agentFunction, the judges score each example’s existing trace. With agentFunction, the agent runs once per example first and the judges score the resulting agent trace.

async function run(testConfig: string | TestConfig, options: OfflineRunOptions = {}): Promise<OfflineTestResult | null>

Parameters

PropType

testConfigstring | TestConfig

Test config name, id, or TestConfig object.

Typestring | TestConfig

options?OfflineRunOptions

Run options (agent function, judge versions, dataset version, pass condition, assert, timeout).

TypeOfflineRunOptions

Default{}

Returns

Promise<OfflineTestResult | null>