Shipping AI That Works: An Evaluation Framework for PMs

Khan explains how AI PM work has changed, from prototyping with AI tools to getting systems ready for production. He compares evals to software tests, while noting that agents are nondeterministic and depend on the data they use. He introduces an LLM-as-a-judge eval as a way to label outputs against a defined goal.

Khan builds a trip planner with multiple agents for budgeting, local experiences, and research, which feed an itinerary agent. Traces show the agents’ inputs, outputs, and actions; a prompt playground lets the team inspect and change prompts. He argues PMs should understand the system and take part in prompt decisions because they own the product outcome.

Khan cautions that a single output cannot show whether a prompt change works at scale. He creates a dataset from 15 synthetic itineraries and compares two prompt versions across 12 examples. LLM-based evals check tone and whether the itinerary offers a discount, while human labels help assess whether those evals are accurate.

Khan says lowering an evaluator model’s temperature can reduce variance, though it cannot eliminate it. In the demo, the judge rates outputs as friendly even when human labels call them robotic, showing that evals need their own checks. He recommends comparing judge results with human labels, refining the eval prompt, and adding examples.

Khan presents eval datasets and criteria as a way to express product requirements and check whether a system is ready to ship. He describes iterating on curated examples during development, then using production data to find new difficult cases. In Q&A, he says the PM’s role includes owning the end experience and helping maintain useful data and evals with the team.