SQL

SQL methods and schema reference for the Judgeval Python SDK.

Query project data with SQL through the Judgeval Python SDK. The virtual schema below abstracts the underlying storage. Judgment validates incoming queries, permits one read-only SELECT, and enforces organization and project scope.

Use sql() to run a query and discover_schema() to fetch the schema. See the SQL guide for setup and authentication.

Schema reference

This reference is generated from the catalog used by SDK schema discovery and MCP discover_schema. It contains no project data.

Judgment SQL (telemetry.v1)

One read-only SELECT against these virtual tables; host supplies org/project scope; SQL cannot override it.

Memory: prefer narrow time ranges or trace/span IDs and only needed fields. Small LIMITs help inspection but do not bound scan, grouping, sort or join memory. Run broad scans sequentially. On QUERY_MEMORY_LIMIT, narrow scope or simplify before retrying.

telemetry.traces

One latest root record per trace_id

Column	Type	Description
`trace_id`	`String`	Trace identifier
`root_span_id`	`String`	Root span identifier
`session_id`	`Nullable(String)`	Session identifier
`started_at`	`DateTime64(6)`	Root span start in UTC
`ended_at`	`Nullable(DateTime64(6))`	Root span end in UTC
`duration_ms`	`Nullable(Float64)`	Root duration in milliseconds
`trace_name`	`String`	Root span name
`status`	`String`	unset, ok, or error
`error_message`	`Nullable(String)`	Root error text
`input`	`Nullable(String)`	Root input
`output`	`Nullable(String)`	Root output
`customer_id`	`Nullable(String)`	Customer identifier
`customer_user_id`	`Nullable(String)`	Customer user identifier
`service_name`	`Nullable(String)`	OTEL service name
`agent_version`	`Nullable(String)`	Agent version
`attributes`	`Map(String, String)`	Span attributes
`resource_attributes`	`Map(String, String)`	OTEL resource attributes

telemetry.spans

One latest record per (trace_id, span_id)

Column	Type	Description
`trace_id`	`String`	Trace identifier
`span_id`	`String`	Span identifier
`parent_span_id`	`Nullable(String)`	Parent span identifier
`session_id`	`Nullable(String)`	Session identifier
`started_at`	`DateTime64(6)`	Span start in UTC
`ended_at`	`Nullable(DateTime64(6))`	Span end in UTC
`duration_ms`	`Nullable(Float64)`	Duration in milliseconds
`span_name`	`String`	Span name
`span_kind`	`String`	OTEL span kind
`service_name`	`Nullable(String)`	OTEL service name
`status`	`String`	unset, ok, or error
`error_message`	`Nullable(String)`	Span error text
`input`	`Nullable(String)`	Span input
`output`	`Nullable(String)`	Span output
`customer_id`	`Nullable(String)`	Customer identifier
`customer_user_id`	`Nullable(String)`	Customer user identifier
`agent_version`	`Nullable(String)`	Agent version
`model`	`Nullable(String)`	Model name
`llm_provider`	`Nullable(String)`	LLM provider
`cost_usd`	`Float64`	Span cost; zero when unreported
`non_cached_input_tokens`	`UInt64`	Non-cached input tokens; zero when unreported
`cache_read_input_tokens`	`UInt64`	Cache-read input tokens; zero when unreported
`cache_creation_input_tokens`	`UInt64`	Cache-creation input tokens; zero when unreported
`output_tokens`	`UInt64`	Output tokens; zero when unreported
`attributes`	`Map(String, String)`	Span attributes
`resource_attributes`	`Map(String, String)`	OTEL resource attributes
`trace_state`	`Nullable(String)`	OTEL trace state
`link_target_trace_id`	`Nullable(String)`	Linked target trace identifier
`link_target_span_id`	`Nullable(String)`	Linked target span identifier
`link_source_trace_id`	`Nullable(String)`	Linked source trace identifier
`link_source_span_id`	`Nullable(String)`	Linked source span identifier
`events_json`	`String`	Ordered span events as JSON objects with Timestamp, Name, and Attributes; [] when empty

telemetry.scores

One retained evaluation result

Column	Type	Description
`score_id`	`String`	Evaluation result identifier
`evaluated_at`	`DateTime64(6)`	Evaluation time in UTC
`evaluation_scope`	`String`	span, trace, session, or unknown
`trace_id`	`Nullable(String)`	Evaluated trace identifier
`span_id`	`Nullable(String)`	Associated span identifier; null when no span is recorded
`session_id`	`Nullable(String)`	Evaluated session identifier
`evaluation_trace_count`	`Nullable(UInt32)`	Number of traces included in a session evaluation; null when unreported
`judge_id`	`String`	Stable judge identifier
`judge_name_at_evaluation`	`String`	Historical judge name
`judge_type_at_evaluation`	`String`	Historical judge type
`judge_major_version`	`UInt32`	Historical major version
`judge_minor_version`	`UInt32`	Historical minor version
`score_type`	`String`	numeric, binary, or categorical
`numeric_value`	`Nullable(Float64)`	Numeric score
`boolean_value`	`Nullable(Bool)`	Binary score
`categorical_value`	`Nullable(String)`	Categorical score
`value_text`	`Nullable(String)`	Normalized score text
`behavior_id_at_evaluation`	`Nullable(String)`	Historical behavior identifier
`reason_json`	`Nullable(String)`	Evaluation reason JSON
`error_message`	`Nullable(String)`	Evaluation error
`customer_id`	`Nullable(String)`	Customer identifier
`customer_user_id`	`Nullable(String)`	Customer user identifier

telemetry.trace_tags

One tag assignment per (trace_id, tag)

Column	Type	Description
`trace_id`	`String`	Tagged trace identifier
`tag`	`String`	Tag value
`created_at`	`DateTime64(6)`	Assignment time in UTC

telemetry.alerts

One alert generated by an automation rule

Column	Type	Description
`alert_id`	`String`	Alert identifier
`trace_id`	`String`	Triggering trace identifier
`rule_id`	`String`	Automation rule identifier
`rule_name`	`String`	Rule name at alert time
`created_at`	`DateTime64(6)`	Alert creation time in UTC
`combine_type`	`String`	Rule condition combination type
`notification_sent`	`Bool`	Whether notification delivery was claimed
`conditions_result_json`	`String`	Rule condition results as JSON
`metadata_json`	`String`	Alert metadata as JSON

evaluations.runs

One current evaluation test run

Column	Type	Description
`run_id`	`String`	Evaluation run identifier
`test_config_id`	`String`	Test configuration identifier
`name`	`String`	Run name
`dataset_id`	`String`	Evaluated dataset identifier
`dataset_version_id`	`String`	Evaluated dataset version identifier
`status`	`String`	Current run status
`source`	`String`	Run initiation source
`created_at`	`DateTime64(6)`	Creation time in UTC
`started_at`	`Nullable(DateTime64(6))`	Start time in UTC
`completed_at`	`Nullable(DateTime64(6))`	Completion time in UTC
`error_message`	`Nullable(String)`	Run failure message

evaluations.items

One latest scorer result per run, example, judge, and judge version

Column	Type	Description
`run_id`	`String`	Evaluation run identifier
`dataset_id`	`String`	Evaluated dataset identifier
`dataset_version_id`	`String`	Evaluated dataset version identifier
`example_id`	`String`	Evaluated example identifier
`offline_trace_id`	`Nullable(String)`	Agent execution trace identifier
`evaluated_at`	`DateTime64(6)`	Evaluation time in UTC
`judge_id`	`String`	Stable judge identifier
`judge_major_version`	`UInt32`	Judge major version
`judge_minor_version`	`UInt32`	Judge minor version
`score_type`	`String`	numeric, binary, or categorical
`numeric_value`	`Nullable(Float64)`	Numeric score
`boolean_value`	`Nullable(Bool)`	Binary score
`categorical_value`	`Nullable(String)`	Categorical score
`value_text`	`Nullable(String)`	Normalized score text
`reason_json`	`Nullable(String)`	Evaluation reason JSON
`metadata_json`	`Nullable(String)`	Evaluation metadata JSON
`success`	`Nullable(Bool)`	Optional test pass-condition result
`error_message`	`Nullable(String)`	Scorer error

datasets.datasets

One current dataset

Column	Type	Description
`dataset_id`	`String`	Dataset identifier
`name`	`String`	Dataset name
`current_version`	`UInt32`	Current version number
`created_at`	`DateTime64(6)`	Creation time in UTC
`updated_at`	`DateTime64(6)`	Last update time in UTC
`schema_json`	`Nullable(String)`	Dataset schema as JSON

datasets.examples

One dataset membership per example, including its version interval

Column	Type	Description
`dataset_id`	`String`	Dataset identifier
`example_id`	`String`	Example identifier
`added_at`	`DateTime64(6)`	Time added to the dataset in UTC
`example_created_at`	`DateTime64(6)`	Example creation time in UTC
`version_added`	`UInt32`	First dataset version containing the example
`version_removed`	`Nullable(UInt32)`	First version excluding the example
`data_json`	`String`	Example data as JSON
`metadata_json`	`String`	User metadata as JSON
`managed_metadata_json`	`String`	Judgment-managed metadata as JSON

config.judges

One judge with a production version per judge_id

Column	Type	Description
`judge_id`	`String`	Stable judge identifier
`name`	`String`	Current judge name
`description`	`Nullable(String)`	Current description
`judge_type`	`String`	Judge implementation type
`score_type`	`String`	Judge score type
`evaluation_mode`	`String`	Production version evaluation mode

config.behaviors

One current behavior value per behavior_id

Column	Type	Description
`behavior_id`	`String`	Stable behavior identifier
`judge_id`	`String`	Owning judge identifier
`value_text`	`String`	Behavior value
`description`	`Nullable(String)`	Behavior description
`created_at`	`DateTime64(6)`	Creation time in UTC
`judge_name`	`String`	Current judge name recorded on the behavior
`categories_json`	`String`	Current category assignments as JSON objects with id, name, and color, ordered by id; [] when none

Offline evidence: use telemetry.offline_traces and telemetry.offline_spans for offline trace IDs, including evaluations.items.offline_trace_id. These have the live tables’ column shapes and payload limits, but never include live rows. Offline telemetry supports trace scope, not live session scope.
Population: traces = root executions; spans = operations; sessions group traces. Count traces from telemetry.traces. State the time range; use all history only when requested. Run broad scans sequentially. Explore cardinality with uniqCombined64; use uniqExact for required exact totals.
Session elapsed_ms = (latest root end - earliest root start), including gaps/overlaps, not summed durations. Aggregate ALL roots before filtering earliest start with inclusive bounds. Usage includes ALL spans with that session ID, even outside the window. Filter IDs in WHERE, grouped metrics in HAVING/outer SELECT. Convert legacy latency_ns thresholds to ms by /1000000. Page by metric plus session_id.
Session search: equality/IN for IDs; ILIKE for substring/prefix/suffix. Behavior filters: rank session scores per (session_id, judge_id) by evaluated_at DESC, score_id DESC; keep latest binary/categorical rows with no error and non-null behavior_id_at_evaluation. Filter sessions via IN, avoiding multiplied usage. Legacy behaviors:any requires ALL requested IDs: group matches by session_id, require count(DISTINCT behavior_id_at_evaluation) = distinct requested ID count.
Sum span cost/tokens once BEFORE tag/score joins. Recorded totals may include duplicate wrapper usage; missing usage = zero, not evidence of free calls.
Scores are results, not unique entities; filter evaluation_scope. Latest behavior inspection ranks by evaluated_at DESC, score_id DESC per entity/judge BEFORE filtering deleted behaviors. Retain errors despite missing metadata; never fall back to older results after deletion/error. History: fetch separately for current-result judge IDs, group by judge_id, paginate by evaluated_at/score_id. Grouped session trace IDs retain duplicates, sorted. value_text is normalized SQL text.
Historical behavior_id_at_evaluation/judge names differ from current config metadata; config.judges contains production versions only. Binary false need not mean failure: interpret rubric/behavior meaning. Keep judge versions separate in offline comparisons. Use dedicated tools for full judge definitions, eligible behavior denominators, health/issues and other absent data; sampled score fractions are not production behavior rates.
Dataset version N: version_added <= N AND (version_removed IS NULL OR version_removed > N). Current = no removal version. LEFT JOIN counts must exclude null example IDs to preserve empty datasets. Evaluation items count scorer results; distinct example IDs count examples.
Separate aggregates from limited detail pages. Order by a unique tie-breaker; continue ORDER BY started_at DESC, trace_id DESC with (started_at < last_time OR (started_at = last_time AND trace_id < last_id)), alongside existing filters.
Attributes: use bounded substringUTF8 windows and lengthUTF8 for total characters. SQL mapContains distinguishes missing keys from present empty strings; direct lookup returns empty for both. No row = missing span; mapKeys lists keys. substringUTF8 uses 1-based Unicode code points: after 1–2000, start at 2001; never reuse byte, JS UTF-16 or tool offsets. Use small literal LIMITs; narrow/page capped results. Replace example IDs with real IDs and include citation IDs.
Use sql_query for trace searches. Calibration review state is not exposed; unreviewed calibration filtering is unavailable.

Examples

Session timing and usage

WITH session_roots AS (
  SELECT session_id, min(started_at) AS started_at, max(ended_at) AS ended_at,
         count() AS trace_count
  FROM telemetry.traces
  WHERE session_id IS NOT NULL
  GROUP BY session_id
), session_usage AS (
  SELECT session_id, sum(cost_usd) AS cost_usd,
         sum(non_cached_input_tokens + cache_read_input_tokens + cache_creation_input_tokens) AS input_tokens,
         sum(output_tokens) AS output_tokens
  FROM telemetry.spans
  WHERE session_id IS NOT NULL
  GROUP BY session_id
)
SELECT r.session_id, r.trace_count, r.started_at, r.ended_at,
       dateDiff('microsecond', r.started_at, r.ended_at) / 1000.0 AS elapsed_ms,
       coalesce(u.cost_usd, 0) AS cost_usd,
       coalesce(u.input_tokens, 0) AS input_tokens,
       coalesce(u.output_tokens, 0) AS output_tokens
FROM session_roots AS r
LEFT JOIN session_usage AS u USING (session_id)
ORDER BY r.started_at DESC, r.session_id DESC
LIMIT 50

Current example counts by dataset

SELECT d.dataset_id, d.name, countIf(e.example_id IS NOT NULL AND e.version_removed IS NULL) AS current_examples
FROM datasets.datasets AS d
LEFT JOIN datasets.examples AS e USING (dataset_id)
GROUP BY d.dataset_id, d.name
ORDER BY current_examples DESC
LIMIT 50