---
title: TestRunsFactory
seo:
  title: TestRunsFactory — Python SDK
  description: Run test suites and read results. Access via client.test_runs. (Python SDK)
description: Run test suites and read results. Access via client.test_runs.
---

A run returns the cases to execute. Run your agent once per case on your
normal tracer with the run and case ids on the trace's root span, using
`Tracer.test_case` or by sending `case["headers"]` to a deployed agent.
Each case is scored as soon as its trace arrives.

```python
run = client.test_runs.start(
    test_suite_id,
    case_filter={"kind": "tags", "tags": ["billing"]},
    candidate=os.environ["GIT_SHA"],
)
for case in run["cases"]:
    with Tracer.test_case(run["run_id"], case["case_id"]):
        support_agent(case["input"])

results = client.test_runs.results(run["run_id"])
```

## start()

Start a run of the test suite's latest published version.

Returns `run_id`, `test_suite_version` and `cases` (each with
`case_id`, `input` and tagging `headers`).

```python
def start(test_suite_id, case_filter=None, candidate=None, candidate_meta=None, trigger=None) -> StartedRun:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `test_suite_id` | `str` | - | |
| `case_filter?` | `Optional[Dict[str, Any]]` | `None` | |
| `candidate?` | `Optional[str]` | `None` | |
| `candidate_meta?` | `Optional[Dict[str, Any]]` | `None` | |
| `trigger?` | `Optional[Literal['manual', 'ci', 'schedule', 'api']]` | `None` | |

### Returns

`StartedRun`

***

## list()

List a test suite's runs, newest first (`runs` and `next_offset`).

```python
def list(test_suite_id, offset=0, limit=50) -> Dict[str, Any]:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `test_suite_id` | `str` | - | |
| `offset?` | `int` | `0` | |
| `limit?` | `int` | `50` | |

### Returns

`Dict[str, Any]`

***

## get()

Get a run and how many of its cases have a trace, are scored and passed.

```python
def get(run_id) -> RunSummary:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `run_id` | `str` | - | |

### Returns

`RunSummary`

***

## results()

Get each case's trace, outcome and per-judge check results.

```python
def results(run_id) -> RunResults:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `run_id` | `str` | - | |

### Returns

`RunResults`

***

## score()

Trigger scoring of an existing run's traces with the latest published version.

Returns the new run's `run_id` right away; its cases are scored in the
background, so poll `get` for progress. The agent doesn't run again.

```python
def score(run_id) -> Dict[str, Any]:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `run_id` | `str` | - | |

### Returns

`Dict[str, Any]`