TestSuitesFactory

Create test suites and edit their cases. Access via client.test_suites.

Cases change only in a suite’s draft (version + 1). New cases go straight into the draft; to edit or remove published cases, call create_draft first. publish freezes the draft as the next version.

suite = client.test_suites.create(name="support-agent-regressions")
client.test_suites.add_cases(
    suite["test_suite_id"],
    [
        {
            "input": {"prompt": "I want a refund"},
            "expected": {
                "<judge-id>": {
                    "judge_major_version": 1,
                    "judge_minor_version": 0,
                    "op": "eq",
                    "expected": "refund",
                }
            },
            "tags": ["billing"],
        }
    ],
)
client.test_suites.publish(suite["test_suite_id"])

create()

Create an empty test suite.

def create(name, description=None, labels=None) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

list()

List the project’s test suites, newest first (test_suites and next_offset).

def list(offset=0, limit=50) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

delete()

Delete a test suite with all its versions, runs and results.

def delete(test_suite_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

list_cases()

List the cases in a published version (default: latest) or the draft.

def list_cases(test_suite_id, version=None, offset=0, limit=100) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

add_cases()

Add cases to the draft. Returns their case_ids in order.

Each case is {"input": ..., "expected": {judge_id: check}, "tags": [...], "name": ..., "metadata": {...}}; only input is required. A check is {"judge_major_version", "judge_minor_version", "op", "expected"}, where op is one of eq, ne, lt, lte, gt, gte, in, not_in.

def add_cases(test_suite_id, cases) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

update_case()

Replace a case in the draft. Call create_draft first for published cases.

def update_case(test_suite_id, case_id, case) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

remove_case()

Remove a case from the draft. Call create_draft first for published cases.

def remove_case(test_suite_id, case_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

create_draft()

Copy the published cases into the draft so they can be edited.

def create_draft(test_suite_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

publish()

Freeze the draft as the next version. Returns the new version.

def publish(test_suite_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

discard()

Throw away the draft’s changes.

def discard(test_suite_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]