Observations

Record each model call your agent makes as an observation: the JSON format and the span attribute that carries it.

An observation records one model call your agent makes: everything the model was given and what it produced. Judgment groups observations into conversations and shows them, exactly as the model saw them, in the Conversation view. Supported integrations record observations automatically. For any other model call, send the observation yourself.

Send an observation

Set the observation on the model call’s span. It travels as JSON text under judgment.conversation.observation. One span holds one observation; setting it again replaces it.

from judgeval import Tracer
from judgeval.trace import ConversationMessage, ConversationObservation

@Tracer.observe(span_type="llm", span_name="support_reply")
def reply(history: list[ConversationMessage]) -> str:
    response = my_model.generate(history)
    observation: ConversationObservation = {
        "model": "my-model-v2",
        "input": [
            {"type": "message", "role": "system", "content": [{"type": "text", "text": "You help customers with orders."}]},
            *history,
        ],
        "output": [
            {"type": "message", "role": "assistant", "content": [{"type": "text", "text": response.text}]},
        ],
        "finish_reason": "stop",
    }
    Tracer.set_conversation_observation(observation)
    return response.text

import { Tracer, type ConversationMessage, type ConversationObservation } from "judgeval";

const reply = Tracer.observe(async (history: ConversationMessage[]) => {
  const response = await myModel.generate(history);
  const observation: ConversationObservation = {
    model: "my-model-v2",
    input: [
      { type: "message", role: "system", content: [{ type: "text", text: "You help customers with orders." }] },
      ...history,
    ],
    output: [
      { type: "message", role: "assistant", content: [{ type: "text", text: response.text }] },
    ],
    finish_reason: "stop",
  };
  Tracer.setConversationObservation(observation);
  return response.text;
}, { spanType: "llm", spanName: "support_reply" });

import json
from opentelemetry import trace

span = trace.get_current_span()
span.set_attribute("judgment.conversation.observation", json.dumps(observation))

Observation

Set conversation_id (or the session ID) to the same value on every call that continues the same context. See Session guidance.

Items

input and output are flat, ordered lists of items. An assistant turn with text and two tool calls is one message item followed by two tool_call items.

Parts

Use custom items and parts for data observations do not define yet.

Usage

Every field is optional. Leave out what you don’t know; don’t send zero.

Each field means the same as the matching judgment.usage.* attribute in Attribute keys.

Errors

error on the observation records a failed model call. error on a tool_result records a failed tool. Both take optional type, message, and stacktrace; {} records a failure without details. Judgment never infers a failure from span status or a missing output.

Rules

• Send the whole window. input holds every item the model was given, including history from earlier calls. Judgment stores each distinct item once per conversation, so resent history costs little storage. It still counts toward the span’s size.
• Keep per-call facts off items. An item must be exactly what is resent on later calls. Finish reason, usage, and latency belong on the observation.
• No unknown fields. Every object is closed. A field this page does not define rejects the whole observation.

Validation and limits

An observation that does not fit this format is not stored. Judgment sets judgment.conversation.warnings on its span, listing each problem by field path. The span and its trace are stored as usual.