Judgment Labs Logo
DocumentationSDK ReferenceInteractive DemoChangelog
Typescript

Tracer

Track agent behavior and evaluate performance in real-time with the Tracer class.

The Tracer class provides comprehensive observability for AI agents and LLM applications. It automatically captures execution traces, spans, and performance metrics while enabling real-time evaluation and monitoring through the Judgment platform.

import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "customer_support_agent",
});

const processQuery = tracer.observe(async function (query: string) {
  const response = await generateLLMResponse(query);
  return response;
}, "llm");

const result = await processQuery("What is machine learning?");

await tracer.shutdown();

All tracing is built on OpenTelemetry standards, ensuring compatibility with the broader observability ecosystem.

client.nodeTracer.create()

Creates a NodeTracer with the specified configuration. Use this for Node.js applications.

async create(config: NodeTracerConfig): Promise<NodeTracer>

Parameters

configrequired

:NodeTracerConfig
Configuration object for the NodeTracer
NodeTracerConfig

projectNamerequired

:string
Name of the project to trace

enableEvaluation

:boolean

Toggle evaluations for asyncEvaluate()

Default: true

enableMonitoring

:boolean

Enable monitoring and metrics collection

Default: false

resourceAttributes

:Record<string, unknown>

OpenTelemetry resource attributes to attach to all spans

Default:

instrumentations

:Instrumentation[]

OpenTelemetry instrumentations to enable (e.g., OpenAIInstrumentation, HttpInstrumentation). Refer to the auto-instrumentation example below.

Default: []

serializer

:Function

Custom serialization function for span data

Default: JSON.stringify

initialize

:boolean

Whether to automatically initialize after creation

Default: true

Returns

Promise<NodeTracer> - Initialized NodeTracer instance

Example

node_tracer.ts
import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "customer_support_agent",
  enableEvaluation: true,
  enableMonitoring: true,
  resourceAttributes: {
    "service.name": "support-api",
    "service.version": "2.0.0",
    "deployment.environment": "production",
  },
});

const myFunction = tracer.observe(async function (input: string) {
  const result = await processData(input);
  return result;
});

await myFunction("test data");
await tracer.shutdown();

Auto-instrumentation

To correctly implement auto-instrumentation on LLM calls, you need to do all of the following:

  1. Initialize an instrumentation file to be preloaded before the application starts.
  2. Import the tracer from the instrumentation file in the application.
  3. Bundle your application using CommonJS.
import { OpenAIInstrumentation } from "@opentelemetry/instrumentation-openai";
import { Judgeval, type NodeTracer } from "judgeval";

export const client = Judgeval.create();

const initPromise = client.nodeTracer
  .create({
    projectName: "auto_instrumentation_example",
    enableEvaluation: true,
    enableMonitoring: true,
    instrumentations: [new OpenAIInstrumentation()],
  })
  .then((t: NodeTracer) => {
    return t;
  });

export async function getTracer(): Promise<NodeTracer> {
  return await initPromise;
}
import { Example } from "judgeval";
import OpenAI from "openai";
import { client, getTracer } from "./instrumentation";

function requireEnv(name: string): string {
  const value = process.env[name];
  if (!value) {
    throw new Error(`Environment variable ${name} is not set`);
  }
  return value;
}

const OPENAI_API_KEY = requireEnv("OPENAI_API_KEY");

const openai = new OpenAI({
  apiKey: OPENAI_API_KEY,
});

async function _chatWithUser(userMessage: string): Promise<string> {
  const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: userMessage },
  ];

  const completion = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages,
  });

  const result = completion.choices[0].message.content || "";

  console.log(`User: ${userMessage}`);
  console.log(`Assistant: ${result}`);

  const tracer = await getTracer(); 

  tracer.asyncEvaluate(
    client.scorers.builtIn.answerRelevancy(),
    Example.create({
      input: "What is the capital of France?",
      actual_output: result,
    })
  );

  return result;
}

(async () => {
  const tracer = await getTracer(); 
  const chatWithUser = tracer.observe(_chatWithUser);

  const result = await chatWithUser("What is the capital of France?");
  console.log(result);

  await new Promise((resolve) => setTimeout(resolve, 10000));
  await tracer.shutdown();
})();

client.browserTracer.create()

Creates a BrowserTracer with the specified configuration. Use this for browser applications.

async create(config: BrowserTracerConfig): Promise<BrowserTracer>

Parameters

configrequired

:BrowserTracerConfig
Configuration object for the BrowserTracer
BrowserTracerConfig

projectNamerequired

:string
Name of the project to trace

enableEvaluation

:boolean

Toggle evaluations for asyncEvaluate()

Default: true

enableMonitoring

:boolean

Enable monitoring and metrics collection

Default: false

resourceAttributes

:Record<string, unknown>

OpenTelemetry resource attributes to attach to all spans

Default:

serializer

:Function

Custom serialization function for span data

Default: JSON.stringify

initialize

:boolean

Whether to automatically initialize after creation

Default: true

Returns

Promise<BrowserTracer> - Initialized BrowserTracer instance

Example

browser_tracer.ts
import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.browserTracer.create({
  projectName: "ecommerce_frontend",
  enableEvaluation: false,
  enableMonitoring: true,
  resourceAttributes: {
    "browser.user_agent": navigator.userAgent,
    "browser.language": navigator.language,
    "app.version": "1.0.0",
  },
});

const handleClick = tracer.observe(async function (event: MouseEvent) {
  const response = await fetch("/api/data");
  const data = await response.json();
  return data;
});

document.getElementById("btn")?.addEventListener("click", handleClick);

observe()

Wraps a function to automatically create spans and record inputs/outputs.

Can also be used as a decorator for class methods. Read more about decorators in the TypeScript handbook.

observe<TArgs extends unknown[], TResult>(
  func: (...args: TArgs) => TResult,
  spanKind?: string
): (...args: TArgs) => TResult

observe(
  spanKind?: string
): (
  target: unknown,
  propertyKey: string | symbol,
  descriptor?: PropertyDescriptor
) => void

Parameters

func

:Function

The function to wrap (automatically provided when used as decorator)

spanKind

:string

Type of span to create. Available options:

  • "span": General span (default)

  • "tool": For functions that should be tracked as agent tools

  • "llm": For language model calls

Default: "span"

Returns

Wrapped function or decorator that creates spans during execution

Examples

import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "document_search",
  enableEvaluation: true,
});

const searchFunction = tracer.observe(async function(query: string) {
  return await searchDatabase(query);
}, "tool");

const results = await searchFunction("search term");
import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "content_generator",
  enableEvaluation: true,
});

class ContentService {
  @tracer.observe("llm")
  async generateResponse(prompt: string): Promise<string> {
    return await callLLM(prompt);
  }

  @tracer.observe("tool")
  async processData(data: unknown): Promise<unknown> {
    return transform(data);
  }
}
const outerFunction = tracer.observe(async function(input: string) {
  const innerFunction = tracer.observe(async function(data: string) {
    return await processData(data);
  }, "tool");

  const result = await innerFunction(input);
  return result;
});

asyncEvaluate()

Asynchronously evaluates a span using a scorer. Read more about scorers in the Evaluation section.

asyncEvaluate(
  scorer: Scorer,
  example: ExampleModel,
): void

Parameters

scorerrequired

:Scorer

The scorer to use for evaluation. Use built-in scorers from client.scorers.builtIn or retrieve prompt scorers

Example: client.scorers.builtIn.answerRelevancy()

examplerequired

:ExampleModel

Example object containing evaluation data with input, actual_output, and optionally expected_output fields

Returns

void - evaluation is triggered asynchronously

Example

async_evaluate.ts
import { Judgeval, Example } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "qa_assistant",
  enableEvaluation: true,
});

const answerQuestion = tracer.observe(async function (input: string) {
  const output = await processInput(input);

  tracer.asyncEvaluate(
    client.scorers.builtIn.answerRelevancy(),
    Example.create({
      input,
      actual_output: output,
      expected_output: "expected",
    }),
  );

  return output;
});

asyncTraceEvaluate()

Asynchronously evaluates an entire trace using a scorer.

asyncTraceEvaluate(scorer: Scorer, model?: string): void

Parameters

scorerrequired

:Scorer
The scorer to use for trace evaluation

model

:string
Model name for evaluation
Default: undefined

Returns

void - evaluation is triggered asynchronously

Example

async_trace_evaluate.ts
import { Judgeval } from "judgeval";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "workflow_orchestrator",
  enableEvaluation: true,
});

const executeWorkflow = tracer.observe(async function () {
  const result = await performComplexOperation();

  tracer.asyncTraceEvaluate(client.scorers.builtIn.faithfulness(), "gpt-4");

  return result;
});

setAttribute()

Sets a single attribute on the current active span.

setAttribute(key: string, value: unknown): void

Parameters

keyrequired

:string
The attribute key

valuerequired

:unknown
The attribute value (will be serialized)

Example

set_attribute.ts
tracer.setAttribute("user.id", "12345");
tracer.setAttribute("request.params", { query: "hello" });

setAttributes()

Sets multiple attributes on the current active span.

setAttributes(attributes: Record<string, unknown>): void

Parameters

attributesrequired

:Record<string, unknown>
Object containing key-value pairs to set

Example

set_attributes.ts
tracer.setAttributes({
  "user.id": "12345",
  "request.method": "POST",
  "request.path": "/api/chat",
});

setSpanKind()

Sets the kind of the current active span.

setSpanKind(kind: string): void

Parameters

kindrequired

:string
The span kind (e.g., 'llm', 'tool', 'span')

Example

set_span_kind.ts
tracer.setSpanKind("llm");

setLLMSpan()

Sets the current span kind to 'llm'.

setLLMSpan(): void

Example

set_llm_span.ts
const processLLMCall = tracer.observe(async function (prompt: string) {
  tracer.setLLMSpan();

  const response = await callOpenAI(prompt);
  return response;
});

setToolSpan()

Sets the current span kind to 'tool'.

setToolSpan(): void

Example

set_tool_span.ts
const searchTool = tracer.observe(async function (query: string) {
  tracer.setToolSpan();
  return await searchDatabase(query);
});

setGeneralSpan()

Sets the current span kind to 'span'.

setGeneralSpan(): void

Example

set_general_span.ts
const processData = tracer.observe(async function (data: unknown) {
  tracer.setGeneralSpan();
  return transform(data);
});

setInput()

Sets the input attribute on the current span.

setInput(input: unknown): void

Parameters

inputrequired

:unknown
The input data to record

Example

set_input.ts
tracer.setInput({ prompt: "What is the capital of France?" });

setOutput()

Sets the output attribute on the current span.

setOutput(output: unknown): void

Parameters

outputrequired

:unknown
The output data to record

Example

set_output.ts
tracer.setOutput({ response: "Paris" });

getConfiguration()

Gets the current tracer configuration.

getConfiguration(): TracerConfiguration

Returns

TracerConfiguration - Current configuration object

getProjectId()

Gets the resolved project ID.

getProjectId(): string | null

Returns

string | null - The project ID or null if not resolved

getSerializer()

Gets the current serializer function.

getSerializer(): Serializer

Returns

Serializer - The serializer function

getSpanExporter()

Gets or creates the span exporter for sending traces to Judgment.

async getSpanExporter(): Promise<JudgmentSpanExporter | NoOpSpanExporter>

Returns

Promise<JudgmentSpanExporter | NoOpSpanExporter> - The span exporter instance

forceFlush()

Flushes all pending spans to the ingest endpoint.

async forceFlush(): Promise<void>

Example

forceFlush.ts
await tracer.forceFlush();

shutdown()

Shuts down the tracer and flushes any pending data.

async shutdown(): Promise<void>

Example

shutdown.ts
await tracer.shutdown();

Judgeval

Run evaluations and manage tracing with the Judgeval class - your primary interface to the Judgment platform.

PromptScorer

Evaluate agent behavior based on a rubric you define and iterate on the platform.

Last updated on

On this page

client.nodeTracer.create()ParametersReturnsExampleAuto-instrumentationclient.browserTracer.create()ParametersReturnsExampleobserve()ParametersReturnsExamplesasyncEvaluate()ParametersReturnsExampleasyncTraceEvaluate()ParametersReturnsExamplesetAttribute()ParametersExamplesetAttributes()ParametersExamplesetSpanKind()ParametersExamplesetLLMSpan()ExamplesetToolSpan()ExamplesetGeneralSpan()ExamplesetInput()ParametersExamplesetOutput()ParametersExamplegetConfiguration()ReturnsgetProjectId()ReturnsgetSerializer()ReturnsgetSpanExporter()ReturnsforceFlush()Exampleshutdown()Example