Judgment Labs Logo
PythonJQL

Pipelines

Python JQL pipeline stages, execution order, and validation limits.

judgeval JQL source on GitHub

Call .pipe() before any select terminal. Stages execute in method-call order and each stage sees only the columns produced by the previous stage.

Pipelines require the traces or spans grain. Calling .pipe() on sessions fails with PIPE_GRAIN.

.where(filter) -> PipelineBuilder
.pick(*, by=None, n=None, per=None, reverse=None) -> PipelineBuilder
.derive(cols) -> PipelineBuilder
.summarize(aggs, *, by=None) -> PipelineBuilder
.sort(by) -> PipelineBuilder
.take(n, offset=None) -> PipelineBuilder
.to_json() -> dict[str, Any]

Stage contract

StageInputs/defaultsOutput and constraints
wherefilterKeeps current rows. After summarize, acts like HAVING.
pickper?, by="timestamp", n=1, reverse=falseKeeps first/last rows per group; adds _rn. 1 <= n <= 1000.
derivecols: name -> expressionAdds columns without changing row count. Requires 1–16 outputs.
summarizeaggs: name -> expression, by?Collapses to one row per group. Requires 1–16 outputs and at most 4 grouping keys.
sortby: stringOrders current rows; append desc for descending.
taken, offset=0Requires n >= 1, offset >= 0.

A pipeline has 1–8 stages in the validated JSON contract.

Stage-filter grammar

Pipeline where and conditional aggregate where do not accept the full root filter grammar. They accept only:

  • eq, ne, gt, gte, lt, lte
  • status
  • all, any, and not composed only from stage filters

Rich filters such as judge, grep, relation quantifiers, and nested source queries belong in the source/root filter before .pipe().

Output column names

derive and summarize output names must match:

^[a-z_][a-z0-9_]{0,63}$

_rn and bucket are reserved. _rn is produced by pick; bucket is produced by a bucket grouping expression.

Example and exact IR

from judgeval.jql import agg_expr, spans

query = (
    spans()
    .last("7d")
    .pipe()
    .summarize({"total_cost": agg_expr("sum", "cost")}, by="model")
    .sort("total_cost desc")
    .take(10)
)
{
  "op": "query",
  "source": "spans",
  "time": { "last": "7d" },
  "pipe": [
    {
      "op": "summarize",
      "by": "model",
      "aggs": {
        "total_cost": { "op": "agg_expr", "func": "sum", "field": "cost" }
      }
    },
    { "op": "sort", "by": "total_cost desc" },
    { "op": "take", "n": 10 }
  ]
}

Referencing a removed or never-produced column fails with UNKNOWN_COLUMN and a hint listing the columns available at that stage.