Complete Beginner's Course on AI Evaluations in 50 Minutes

Khan describes code-based pass-or-fail checks, human reviews, LLM-as-a-judge evaluations, and feedback from real users. Evals measure AI system quality and help teams catch hallucinations before they harm the customer experience.

Using an On running shoes support bot, they define criteria for product knowledge, policy compliance, and tone. They test sample questions, label the answers in a spreadsheet, and debate borderline cases, including what the bot should do when the return policy does not cover a missing shoe box.

One answer incorrectly says that 45 minutes exceeds a 60-minute cancellation window, illustrating how an agent can misread a policy. Khan recommends starting with about 10 examples for internal testing and roughly 100 or more when seeking greater confidence for production. The team uses human labels and notes to guide prompt changes, then checks whether the new answers improve.

They move the spreadsheet into Arize, replay the same five questions with another model, and use an LLM judge to score product knowledge, policy compliance, and tone. They compare judge scores with human labels: product knowledge matches 100% in their small sample, while tone matches only once. Khan says to verify the judge on a handful of examples before relying on it at larger scale.

The process moves from a basic prompt and evaluation criteria to manual labeling, repeated iteration, and then checking an LLM judge against human judgment. They suggest beginning internally, then testing with a small share of traffic, such as 1% or 10%, and examining both user feedback and business results. A thumbs-down can signal dissatisfaction without identifying what the system did wrong, so teams need to inspect the underlying examples.