---
title: "Trace durable work"
description: "Trace long-running loops, queued workers, workflow engines, and approval waits."
seo:
  title: "Durable AI Agent Tracing | Judgment Docs"
  description: "Trace queued jobs, workflow activities, retries, and human approval waits in Judgment using separate traces in one session."
---

Durable work uses one trace for each autonomous part and one session for the
full job, because a root span cannot stay open across a queue, checkpoint,
process stop, or human approval wait. Use this page for checkpointed loops,
queues, workflow engines, worker replacement, and human approval waits.

Start a new trace when each autonomous part begins, and end it before work
crosses one of those boundaries. Keep the same session ID for every part of the
job.

## Trace the action that starts work

A job passes through many hands between submission and completion, and most of
them only move it along. Trace a handler when the work itself starts there, and
skip it when the job just passes through.

A request or command that drops the job on a queue and returns is a
pass-through: the work starts later, in the worker, and the worker's trace is
the first one in the session. A handler that starts the run in its own process,
or restarts one after a crash, is different. The work begins there, so that
handler gets its own root in the job's session. The process boundary decides,
not the return time. A handler that starts a background loop and returns
immediately still starts the work in its own process. Record the job input on
it and end it before the job waits.

The rest follows the same test. A worker receiving a queue delivery starts a
new trace in the same session. An approval handler that only acknowledges the
event gets no trace, and neither do health checks or status polls.

## Long-running loops

Each iteration or step can be one durable part. Keep every part of the run in
the same session.

End and flush the current part before saving its checkpoint, even when the loop
runs inside a long-running server. An unclean kill then loses at most the
in-flight part, never finished work sitting in the export queue.

## Queues and worker processes

Pass the job or session ID in the task message or saved state. The worker
starts a root when it receives the job and sets that ID as the session.

```python
from judgeval import Tracer

task = {"job_id": job_id, "payload": payload}
queue.send(task)


@Tracer.observe(span_type="agent", span_name="job.run")
def process_task(task: dict):
    Tracer.set_session_id(task["job_id"])
    return run(task["payload"])
```

An automatic retry remains part of the same job: each attempt is its own trace
in the same session, with the attempt number as an attribute. Whether the failed
attempt leaves a trace depends on how the worker died. A graceful stop (SIGTERM
with time to flush) exports the failed attempt as an error trace. A hard kill
loses the unfinished root, so the retry may be the only trace for that step. If
the failed attempt is missing after a hard kill, that is expected. Buffering or
delaying the work to force it into the export queue would change the app's
behavior.

Use [Distributed tracing](/documentation/tracing/distributed) only
when services must share trace context for one live request. A queue delivery
or approval wait is not one live request. Work that resumes later starts a new
trace in the same session.

## Workflow engines (Temporal and similar)

Workflow code replays, so it cannot own a live app root. Trace the
activities instead:

- One trace per activity execution, created by the activity itself.
- Every trace carries the workflow or job ID as its session ID.
- Record the attempt number as an attribute so retries are searchable.
- Initialize the tracer at import time in each process that creates spans: the
  producer and every worker.
- Mark handled failures on the span that returned them. A cancelled or
  timed-out attempt is a failure: mark it as an error, not a clean trace with
  empty output.
- End and flush the activity root in the activity's outer `finally` block
  before the activity returns.

```python title="activities.py"
from judgeval import Tracer
from temporalio import activity

import app.tracing  # module that calls Tracer.init() at import time


@Tracer.observe(span_type="agent", span_name="app.execute_step")
async def _execute_traced_step(req: StepRequest) -> StepResult:
    info = activity.info()
    Tracer.set_session_id(info.workflow_id)
    Tracer.set_attributes({"app.activity.attempt": info.attempt})
    return await run_step(req)   # model and tool calls happen here


@activity.defn
async def execute_step(req: StepRequest) -> StepResult:
    try:
        return await _execute_traced_step(req)
    finally:
        Tracer.force_flush()
```

The submission root ends when the producer request returns, and each activity
starts its own trace. Adding a manual span for an operation that an integration
already records would duplicate it. The trace boundaries should follow the
app's existing activities, retries, and heartbeats. See the
[Temporal integration](/documentation/integrations/other/temporal) page before
enabling Temporal's own tracing interceptor.

## Human approval

End the current trace before the app waits. The approval request or signal
gets a root in the same job session only when its handler does app work, such
as resuming the run in its own process.
Record fields you will search for, such as the approver and phase, as
attributes. If the handler only records the approval, the next autonomous part
starts the new root. Approval status polls do not start work, so they do not
need spans.

## Verify

First run the
[shared verification checklist](/documentation/tracing/instrumentation#verify-before-you-claim-success).
Then kill a worker mid-activity, let the engine retry, and resume the workflow.
The activity traces should stay in the same session, and the retry should have a
higher attempt number. A failed attempt may be missing after a hard kill. Stop
the worker gracefully when you need to verify that attempt as an error trace.

## Troubleshooting

| Symptom | Check |
| --- | --- |
| One trace stays open across a queue or approval | End and flush the current part before the wait, then start a new root when work resumes. |
| Retries appear in unrelated sessions | Pass the stable workflow or job ID and set it on every attempt's root. |
| Completed work disappears after a worker stops | Flush the completed root before returning or saving the checkpoint. |
| Workflow replay duplicates traces | Trace activities, not replayable workflow code. |

## Related pages

- [Tracing data model](/documentation/tracing)
- [Distributed tracing](/documentation/tracing/distributed) for one live request
  crossing services
- [Trace subagents](/documentation/tracing/subagents)