37:44Nick Ung and Akshay Sharma describe how Lyft evaluates customer support AI agents before and after launch. They recommend realistic offline simulations based on production data, user personas, and difficult conversations, paired with deterministic checks and task-specific LLM judges. They explain how to calibrate judges against human labels, track uncertainty in reported scores, and use trace review and error analysis to improve agents continuously. They also discuss building a repeatable, config-driven evaluation harness that contributors can use across development and deployment gates.

