Judgeval

The Judgeval class is your primary interface for interacting with the Judgment platform. It provides access to tracing, scoring, and evaluation functionality through a unified API.

import { Judgeval } from "judgeval";

const client = Judgeval.create(); 

const clientWithConfig = Judgeval.create({
  apiKey: process.env.JUDGMENT_API_KEY,
  organizationId: process.env.JUDGMENT_ORG_ID,
});

Authentication

Set up your credentials using environment variables:

export JUDGMENT_API_KEY="your_key_here" 
export JUDGMENT_ORG_ID="your_org_id_here"

# Add to your .env file
JUDGMENT_API_KEY="your_key_here"
JUDGMENT_ORG_ID="your_org_id_here"

Judgeval.create()

Initialize a Judgeval client instance.

static create(config?: JudgevalConfig): Judgeval

If no configuration is provided, the Judgeval client will automatically use the JUDGMENT_API_KEY and JUDGMENT_ORG_ID environment variables.

Parameters

config:JudgevalConfig

Optional configuration object

Default: undefined

JudgevalConfig

apiKey:string

Your Judgment API key.

Recommended: set with JUDGMENT_API_KEY environment variable.

Default: process.env.JUDGMENT_API_KEY

organizationId:string

Your organization ID.

Recommended: set with JUDGMENT_ORG_ID environment variable.

Default: process.env.JUDGMENT_ORG_ID

apiUrl:string

API URL for custom endpoints.

Recommended: set with JUDGMENT_API_URL environment variable.

Default: process.env.JUDGMENT_API_URL

Throws

Error if API key or organization ID are not found

Example

client.ts

import { Judgeval } from "judgeval";

const client = Judgeval.create();

const clientWithConfig = Judgeval.create({
  apiKey: "your-api-key",
  organizationId: "your-org-id",
});

Accessing Features

The Judgeval client provides access to various features through factory properties:

client.nodeTracer

Access the Node.js tracer factory for creating tracers with auto-instrumentation.

client.nodeTracer.create(config: NodeTracerConfig): Promise<NodeTracer>

See Tracer for detailed documentation.

client.browserTracer

Access the browser tracer factory for creating tracers in browser environments.

client.browserTracer.create(config: BrowserTracerConfig): Promise<BrowserTracer>

client.scorers

Access scorer factories for creating and retrieving scorers.

client.scorers.builtIn.answerRelevancy();
client.scorers.builtIn.answerCorrectness();
client.scorers.builtIn.faithfulness();
client.scorers.promptScorer;
client.scorers.tracePromptScorer;
client.scorers.customScorer;

See PromptScorer for detailed documentation.

client.evaluation

Access the evaluation factory for running batch evaluations.

client.evaluation.create(config?: EvaluationConfig): Evaluation

Complete Usage Example

Here's a complete example showing how to use the new Judgeval API with tracing and evaluation:

complete_example.ts

import { Judgeval, Example } from "judgeval";
import { OpenAIInstrumentation } from "@opentelemetry/instrumentation-openai";
import OpenAI from "openai";

const client = Judgeval.create();

const tracer = await client.nodeTracer.create({
  projectName: "customer_support_agent",
  enableEvaluation: true,
  enableMonitoring: true,
  instrumentations: [new OpenAIInstrumentation()],
});

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function _chatWithUser(userMessage: string): Promise<string> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: userMessage }],
  });

  const result = response.choices[0].message.content || "";

  tracer.asyncEvaluate(
    client.scorers.builtIn.answerRelevancy(),
    Example.create({
      input: userMessage,
      actual_output: result,
    }),
  );

  return result;
}

const chatWithUser = tracer.observe(_chatWithUser);

const result = await chatWithUser("What is the capital of France?");
console.log(result);

await tracer.shutdown();

Judgeval

On this page