Durable AI Agent Tracing

Durable work uses one trace for each autonomous part and one session for the full job, because a root span cannot stay open across a queue, checkpoint, process stop, or human approval wait. Use this page for checkpointed loops, queues, workflow engines, worker replacement, and human approval waits.

Start a new trace when each autonomous part begins, and end it before work crosses one of those boundaries. Keep the same session ID for every part of the job.

Trace the action that starts work

A job passes through many hands between submission and completion, and most of them only move it along. Trace a handler when the work itself starts there, and skip it when the job just passes through.

A request or command that drops the job on a queue and returns is a pass-through: the work starts later, in the worker, and the worker's trace is the first one in the session. A handler that starts the run in its own process, or restarts one after a crash, is different. The work begins there, so that handler gets its own root in the job's session. The process boundary decides, not the return time. A handler that starts a background loop and returns immediately still starts the work in its own process. Record the job input on it and end it before the job waits.

The rest follows the same test. A worker receiving a queue delivery starts a new trace in the same session. An approval handler that only acknowledges the event gets no trace, and neither do health checks or status polls.

Long-running loops

Each iteration or step can be one durable part. Keep every part of the run in the same session.

End and flush the current part before saving its checkpoint, even when the loop runs inside a long-running server. An unclean kill then loses at most the in-flight part, never finished work sitting in the export queue.

Queues and worker processes

Pass the job or session ID in the task message or saved state. The worker starts a root when it receives the job and sets that ID as the session.

from judgeval import Tracer

task = {"job_id": job_id, "payload": payload}
queue.send(task)


@Tracer.observe(span_type="agent", span_name="job.run")
def process_task(task: dict):
    Tracer.set_session_id(task["job_id"])
    return run(task["payload"])

An automatic retry remains part of the same job: each attempt is its own trace in the same session, with the attempt number as an attribute. Whether the failed attempt leaves a trace depends on how the worker died. A graceful stop (SIGTERM with time to flush) exports the failed attempt as an error trace. A hard kill loses the unfinished root, so the retry may be the only trace for that step. If the failed attempt is missing after a hard kill, that is expected. Buffering or delaying the work to force it into the export queue would change the app's behavior.

Use Distributed tracing only when services must share trace context for one live request. A queue delivery or approval wait is not one live request. Work that resumes later starts a new trace in the same session.

Workflow engines (Temporal and similar)

Workflow code replays, so it cannot own a live app root. Trace the activities instead:

One trace per activity execution, created by the activity itself.
Every trace carries the workflow or job ID as its session ID.
Record the attempt number as an attribute so retries are searchable.
Initialize the tracer at import time in each process that creates spans: the producer and every worker.
Mark handled failures on the span that returned them. A cancelled or timed-out attempt is a failure: mark it as an error, not a clean trace with empty output.
End and flush the activity root in the activity's outer finally block before the activity returns.

activities.py

from judgeval import Tracer
from temporalio import activity

import app.tracing  # module that calls Tracer.init() at import time


@Tracer.observe(span_type="agent", span_name="app.execute_step")
async def _execute_traced_step(req: StepRequest) -> StepResult:
    info = activity.info()
    Tracer.set_session_id(info.workflow_id)
    Tracer.set_attributes({"app.activity.attempt": info.attempt})
    return await run_step(req)   # model and tool calls happen here


@activity.defn
async def execute_step(req: StepRequest) -> StepResult:
    try:
        return await _execute_traced_step(req)
    finally:
        Tracer.force_flush()

The submission root ends when the producer request returns, and each activity starts its own trace. Adding a manual span for an operation that an integration already records would duplicate it. The trace boundaries should follow the app's existing activities, retries, and heartbeats. See the Temporal integration page before enabling Temporal's own tracing interceptor.

Human approval

End the current trace before the app waits. The approval request or signal gets a root in the same job session only when its handler does app work, such as resuming the run in its own process. Record fields you will search for, such as the approver and phase, as attributes. If the handler only records the approval, the next autonomous part starts the new root. Approval status polls do not start work, so they do not need spans.

Verify

First run the shared verification checklist. Then kill a worker mid-activity, let the engine retry, and resume the workflow. The activity traces should stay in the same session, and the retry should have a higher attempt number. A failed attempt may be missing after a hard kill. Stop the worker gracefully when you need to verify that attempt as an error trace.

Durable work