TestRunsFactory

Run test suites and read results. Access via client.test_runs.

A run returns the cases to execute. Run your agent once per case on your normal tracer with the run and case ids on the trace’s root span, using Tracer.test_case or by sending case["headers"] to a deployed agent. Each case is scored as soon as its trace arrives.

run = client.test_runs.start(
    test_suite_id,
    case_filter={"kind": "tags", "tags": ["billing"]},
    candidate=os.environ["GIT_SHA"],
)
for case in run["cases"]:
    with Tracer.test_case(run["run_id"], case["case_id"]):
        support_agent(case["input"])

results = client.test_runs.results(run["run_id"])

start()

Start a run of the test suite’s latest published version.

Returns run_id, test_suite_version and cases (each with case_id, input and tagging headers).

def start(test_suite_id, case_filter=None, candidate=None, candidate_meta=None, trigger=None) -> StartedRun:

Parameters

Returns

StartedRun

list()

List a test suite’s runs, newest first (runs and next_offset).

def list(test_suite_id, offset=0, limit=50) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]

get()

Get a run and how many of its cases have a trace, are scored and passed.

def get(run_id) -> RunSummary:

Parameters

Returns

RunSummary

results()

Get each case’s trace, outcome and per-judge check results.

def results(run_id) -> RunResults:

Parameters

Returns

RunResults

score()

Trigger scoring of an existing run’s traces with the latest published version.

Returns the new run’s run_id right away; its cases are scored in the background, so poll get for progress. The agent doesn’t run again.

def score(run_id) -> Dict[str, Any]:

Parameters

Returns

Dict[str, Any]