The AI Skill That Will Define Your PM Career in 2025

Khan sees AI as becoming part of every product manager’s skill set, whether they build with models or use AI tools in their work. He recommends learning what different approaches can do, rather than treating large language models as the answer to every problem. For example, products may combine an LLM with ranking or classification models.

Khan names customer curiosity, prototyping, and measurement as core product skills alongside AI fundamentals. He describes building a customer usage dashboard prototype during a meeting with Replit, using Pendo documentation as input. Evals and observability then help teams measure how well a system performs, rather than simply shipping it.

Khan defines evals as ways to measure how well a product turns user input into a useful outcome, especially when LLM behavior is variable. Writing them requires understanding likely user scenarios and expressing expectations in writing. He recommends grouping real scenarios and breaking systems into smaller components so teams can evaluate and iterate more tightly.

For a retail support agent handling laptop returns, Khan suggests starting with real user inputs and human-labeled examples, such as whether the customer is within the return window. Teams can add simple checks for rule violations, abusive language, or jailbreak attempts, then use initial examples to generate more test cases. As the product develops, teams can test prompts, tools, and other components with beta users and compare results against their evaluations.

Khan says an eval can be a prompt that asks a judge model to label another system’s answer as correct or incorrect. Examples in the prompt can improve results, and he advises prioritizing quality before optimizing model costs. He encourages aspiring AI product managers to try tools themselves, build prototypes, and learn with people working on AI products.