---
title: EvaluatorRunner
seo:
  title: EvaluatorRunner — Python SDK
  description: Abstract base for evaluation runners. (Python SDK)
description: Abstract base for evaluation runners.
---

| Prop | Type | Default | Description |
| - | - | - | - |
| `S?` | `Any` | `TypeVar('S', str, Judge)` | |
| `ExperimentRunItem?` | `Any` | `Dict[str, Any]` | |

Abstract base for evaluation runners.

Concrete implementations handle either hosted (server-side) or local
(in-process) scorer execution.  The generic parameter `S` is `str`
for hosted scorers or `Judge` for local scorers.

## \_\_init\_\_()

```python
def __init__(client, project_id, project_name):
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `client` | `JudgmentSyncClient` | - | |
| `project_id` | `Optional[str]` | - | |
| `project_name` | `str` | - | |

***

## run()

Execute an evaluation run and return results.

```python
def run(examples, scorers, eval_run_name, assert_test=False, timeout_seconds=300) -> typing.List:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `examples` | `List[Example]` | - | Examples to evaluate. |
| `scorers` | `List[S]` | - | Scorers to run (strings or Judge instances). |
| `eval_run_name` | `str` | - | Name for this evaluation run. |
| `assert_test?` | `bool` | `False` | Deprecated and ignored by the current evaluation result payload. |
| `timeout_seconds?` | `int` | `300` | Maximum time to wait for results. |

### Returns

`typing.List` - A list of `ScoringResult` objects, one per example.

***

## \_binary\_label()

```python
def _binary_label(value) -> str:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `value` | `bool` | - | |

### Returns

`str`

***

## \_scorer\_value()

```python
def _scorer_value(scorer_dict) -> str | float | None:
```

### Parameters

| Prop | Type | Default | Description |
| - | - | - | - |
| `scorer_dict` | `Mapping[str, Any]` | - | |

### Returns

`str | float | None`