Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

Dat distinguishes observability, which shows what a system is doing, from evals, which help teams identify what is working. Platform teams may focus on cost and latency, while application teams may focus on product quality. He describes evals as signals that make large volumes of traces easier to assess.

Dat says LLM-as-a-judge is only one tool. Teams can also use human feedback, golden datasets, smaller encoder models, and code checks such as keyword, regex, or JSON validation. A golden dataset can help measure and tune whether an LLM judge matches trusted human labels.

One loop uses traces and eval results to find failures, annotate examples, and improve prompts, models, or orchestration. A second loop checks whether the evals themselves are accurate and tunes them when their judgments are wrong or too vague. Dat argues that faster iteration can improve product quality, giving four monthly iterations versus two as an example.

Evals can target a single model call, a larger workflow, control flow, or a full conversation session. Dat recommends conditional evals when a failed earlier step makes later checks wasteful. For agents, aggregate path graphs can reveal failure patterns across traces, while trajectory evals compare an expected route with the components the agent actually used.

Dat frames evals as either inside orchestration or outside it, with tradeoffs such as latency and complexity. Guardrails can reduce risk, but he says teams should also fix the underlying prompts and application logic; guardrails are not infallible. He describes OpenTelemetry propagation for tracing work across services and meta-prompting to use eval results and examples to suggest improved prompts.