Subprocess models

When an app calls a model through a CLI or subprocess, Judgment needs one llm child span for each invocation because an integration cannot see inside the subprocess. If the app calls a provider SDK directly, use the provider's integration instead.

Record each subprocess invocation as an llm child:

1. Keep the app root open through the call, parsing, and result saving.
2. Parse the result and call recordLLMMetadata() with model, provider, non_cached_input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, and total_cost_usd.
3. Set cost only when the subprocess reports it. Recording 0 for an unknown cost makes a paid session look free and makes the total unreliable.
4. Mark a non-zero exit, error payload, or denial as an error, even when the process exits cleanly.
5. Record useful values such as duration, time to first token, turn count, and stop reason as separate attributes so they can be queried without opening the raw result.

from judgeval import Tracer
from opentelemetry.trace import Status, StatusCode

@Tracer.observe(span_type="llm", span_name="cli.model")
def call_model(prompt: str) -> dict:
    completed = run_cli(prompt)
    parsed = parse_cli_output(completed.stdout)

    metadata = {
        "model": parsed.model,
        "provider": parsed.provider,
        "non_cached_input_tokens": parsed.input_tokens,
        "output_tokens": parsed.output_tokens,
    }
    if parsed.cost_usd is not None:
        metadata["total_cost_usd"] = parsed.cost_usd

    Tracer.recordLLMMetadata(metadata)
    if completed.returncode != 0 or parsed.is_error:
        Tracer.get_current_span().set_status(
            Status(StatusCode.ERROR, completed.stderr or "CLI call failed")
        )

    return parsed.result

Use the subprocess conversation ID

When the subprocess owns the conversation, use its conversation ID as the session ID. Keep the root active until you parse and set that ID. Save your app's request or wrapper ID as a separate attribute.

When the app resumes the conversation, pass the same subprocess ID back to the CLI and use it again as the session ID.

Claude Code structured output

Read model IDs from the keys in modelUsage; one call may use more than one model. Check is_error, api_error_status, and permission_denials even when the exit code is zero.

Verify

First run the shared verification checklist. Then run one real task and open the llm span. It should show the reported model, tokens, and cost. If the CLI reported a cost, the session total should not be $0.00.

Related pages

• Record each LLM call once
• Durable work for subprocesses that run inside queued jobs
• Attribute Keys for the judgment.* attributes stored on each span