Log feedback
Submit feedback for a trace, traced session, or a recorded conversation/trajectory.
Attach user ratings or scores from your own evaluators to traces, sessions, or recorded conversations in Judgment. For feedback on one response, save its trace ID while handling the request, then look it up using your application's message ID when feedback arrives. Use a session ID when the rating describes an entire traced session. For a trajectory recorded without traces, use its conversation ID as shown below.
You need SDK credentials
and a project. For trace and session feedback, enable
tracing. For conversation feedback,
record a trajectory as shown below; tracing is not required. The examples use
my-project; record the target and submit feedback in the same project. The
target must already be visible in Judgment before you submit feedback.
Create the judges once
An external judge defines the values your application can submit. This example uses a boolean for response helpfulness and a number for session quality.
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
client.external_judges.create(name="Response helpful", score_type="binary")
client.external_judges.create(name="Session quality", score_type="numeric")Capture and retain the trace ID during the request
Give each response that can receive feedback a unique message ID. Inside the observed request, get the active span's trace ID and save the mapping before returning the response. Return the message ID to your application so a later feedback request can identify the response.
These examples use a local JSON file so the mapping survives process exit. Use your application's database in production, scoped to your organization and project, with a unique constraint on the message ID. The file example is for a single process at a time; it does not coordinate concurrent workers. Do not overwrite a message's saved trace ID when another request runs.
import json
from pathlib import Path
from judgeval import Tracer
tracer = Tracer.init(project_name="my-project")
if not tracer.project_id:
raise RuntimeError("Tracing requires valid credentials and an existing project")
targets_file = Path("feedback-targets.json")
@Tracer.observe(span_type="agent")
def answer(message_id: str, session_id: str):
# Attach your existing conversation/session ID to each request in the session.
Tracer.set_session_id(session_id)
span = Tracer.get_current_span()
if not span.is_recording():
raise RuntimeError("Enable tracing before retaining a feedback target")
# Keep all 32 hexadecimal characters, including leading zeros.
trace_id = f"{span.get_span_context().trace_id:032x}"
targets = json.loads(targets_file.read_text()) if targets_file.exists() else {}
if message_id in targets:
raise ValueError("Use a unique message ID for each response being rated")
targets[message_id] = {"project_name": "my-project", "trace_id": trace_id}
targets_file.write_text(json.dumps(targets))
return {"message_id": message_id, "answer": "Here is the answer."}
response = answer("message-123", "conversation-456")
if not Tracer.force_flush():
raise RuntimeError("Trace export did not finish")
print(response)Initialization must resolve the project before you save feedback targets. A recording span alone does not guarantee Judgment export is enabled. Capture the ID while the request's span is active. A later feedback worker has no active request span to consult. The returned response contains your message ID; keep the trace mapping on your server and authorize feedback against that message.
Look up the trace when feedback arrives
This code can run in a separate process after the traced request has finished. It uses the customer's message ID, without searching Judgment for matching attributes or relying on an active tracing context.
import json
from pathlib import Path
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
message_id = "message-123" # From your authorized feedback request.
target = json.loads(Path("feedback-targets.json").read_text())[message_id]
if target["project_name"] != "my-project":
raise ValueError("Feedback target belongs to a different project")
client.external_judges.submit_result(
judge_name="Response helpful",
target={"type": "trace", "trace_id": target["trace_id"]},
value=False,
reason="The user said this response did not help.",
)Open that trace in Judgment and confirm the response-helpfulness result appears.
Exporting spans and making them queryable are separate steps: even after
force_flush / forceFlush, ingestion may still be in progress. If submission
returns a target-not-found error, wait for the trace to become visible and retry
with the same target. Do not substitute a session ID or the first search match.
When to use a session ID
Use a trace target for a rating of one response or run. Use a session target for
an overall conversation rating, resolution outcome, or session-wide evaluator.
Attach your existing conversation ID using set_session_id / setSessionId to
every traced request in that session, as shown above. Judgment then knows that
customer-owned ID as a session ID. A session target rates the session; it does
not apply a separate rating to every trace in it.
Once that session is visible, submit its ID directly. No trace lookup or retained message mapping is needed for a whole-session rating:
client.external_judges.submit_result(
judge_name="Session quality",
target={"type": "session", "session_id": "conversation-456"},
value=4.5,
reason="Overall rating after the conversation ended.",
)Use trace for one execution, session for a traced session, or conversation
for a recorded trajectory (see below). An ID from another system
becomes a session target only after you attach it to your traces as a session ID;
supplying an arbitrary conversation ID does not create a session.
Rate a trajectory without traces
A trajectory recorded with Trajectory.record uses your conversation_id.
Submit a conversation target to rate that whole trajectory. This is separate
from a traced session: conversation feedback works even when no trace or session
was recorded. The conversation must be visible in Judgment before submission.
Create a numeric judge once, then record the conversation using your existing ID:
from judgeval import Judgeval, Trajectory
client = Judgeval(project_name="my-project")
client.external_judges.create(name="Conversation quality", score_type="numeric")
Trajectory.init(project_name="my-project")
Trajectory.record({
"conversation_id": "conversation-789",
"input": [{"type": "message", "role": "user", "content": [{"type": "text", "text": "Help me track my order."}]}],
"output": [{"type": "message", "role": "assistant", "content": [{"type": "text", "text": "Your order arrives tomorrow."}]}],
})
Trajectory.flush()Once the conversation appears in Judgment, a later feedback worker needs only that conversation ID. It does not need a retained trace-ID mapping:
from judgeval import Judgeval
client = Judgeval(project_name="my-project")
client.external_judges.submit_result(
judge_name="Conversation quality",
target={"type": "conversation", "conversation_id": "conversation-789"},
value=4.5,
reason="Overall rating of the recorded trajectory.",
)Open the conversation's results panel to see the customer score. You can also
query conversations.scores and filter source = 'customer' using
SQL. Judgment-generated scores
remain separate. Resubmitting replaces the same judge's current customer score
for that conversation. This target rates the whole conversation; it does not
assign individual message/item scores.
Values, corrections, and retries
Values must match the judge's production version: a boolean for binary, a
finite Float32-compatible number for numeric, or a configured choice name for
categorical. You can also submit customer results under an existing internal
judge; its Judgment-generated results remain separate.
Judgment keeps one current customer score per judge and target. Submitting again for the same judge and trace, session, or conversation replaces the previous customer score. Previous values are not retained as a review history. A retry submits its value again and can replace a correction submitted in the meantime. Submitted results do not execute a judge or trigger judge actions.
For complete runnable commands, see the Python example and TypeScript example. For accepted parameters and return values, see the Python reference or TypeScript reference.