Skip to content
Judgment Labs
Esc
↑↓navigate↵open⌘Jpreview
On this page

The example agent

The lunch-order agent that the docs use as a running example.

The guides and cookbooks in these docs all use the same agent, so you can follow one set of traces from setup to testing.

The lunch-order agent is an agent teams use to order lunch to their office. Someone types a request like “order me mediterranean food to feed 30 hungry engineers”, and the agent finds restaurants nearby, checks that they can cater a group that size, prices the order, and asks the user to confirm before it places it.

Its Judgment project is called lunch-order-agent.

Its tools

Tool What it does
get_user_context Looks up the office address, past orders, and dietary notes.
search_nearby_restaurants Finds restaurants by cuisine and distance.
web_search Searches the web when the restaurant data is thin.
get_menu Fetches a restaurant’s menu from a third-party API.
check_catering_capacity Checks whether a restaurant can feed the headcount.
get_reviews Pulls ratings for a restaurant.
estimate_order Prices the order.
check_delivery_eta Estimates when the food arrives.

One trace

Here’s what one run looks like in Judgment, simplified:

  • User request: “order me mediterranean food to feed 30 hungry engineers”
    • llm call: understand the request (cuisine: mediterranean, headcount: 30)
    • tool get_user_context: office address, past orders, dietary notes
    • tool search_nearby_restaurants: 9 mediterranean restaurants within 2 miles
    • llm call: shortlist restaurants that offer catering (4 left)
    • tool get_menu x3: Olive & Thyme and Falafel House return menus, Cedar Grill fails (504 Gateway Timeout)
    • tool get_menu retry: Cedar Grill fails again
    • llm call: drop Cedar Grill and continue with the other two
    • tool check_catering_capacity x2: Olive & Thyme can handle 30 people, Falafel House can’t
    • tool get_reviews: Olive & Thyme has 4.7 stars
    • tool estimate_order: $612 total
    • tool check_delivery_eta: 12:15 PM
    • Response: “I recommend ordering from Olive & Thyme (0.4 mi away): 8 mezze platters, 30 chicken shawarma bowls (6 vegetarian), and 3 sides of baklava. $612 total, delivered by 12:15 PM. Want me to place the order?”

When the user replies “yes, place it”, that’s a second trace in the same session.

A real trace has a lot more in it: raw search results, full menus, prompts, token counts. That’s what makes traces useful and also what makes them hard to read at scale.

What goes wrong

The response above looks fine, but Cedar Grill was dropped without the user ever knowing. Root-cause an Issue starts from the Issue Judgment finds for that failure and ends with a judge that watches for it.

The other cookbooks use different parts of the same agent:

  • Alerting alerts on jailbreak attempts, cost and latency, and reuses the Root-cause judge to send Failed silently to Slack.
  • Data analysis adds judges for user sentiment, order size, cuisine and completed requests, then asks why users got frustrated last week.
  • Testing compares lunch-agent-v1, which ignores budgets, with a lunch-agent-v2 that respects them, over a dataset of requests like “order vegetarian lunch for 12 under $200”.

Was this page helpful?