Real ROI: Lessons from Enterprises that have already succeeded with LLMs at Scale

Habib says companies have moved from experimentation to revenue and cost savings. He cites legal software company Filevine, which launched six products in a year and roughly doubled its revenue. He describes most LLM applications as a base model, prompt, data selection strategy, and sometimes function calling, with the work focused on improving each component.

Habib recommends starting with people, then ideas, then tools. Successful teams tend to have many generalist product engineers, domain experts who help shape prompts and judge results, and some machine learning expertise, especially for sound evaluation practices. At Duolingo, linguists handled prompt engineering; at Filevine, legal professionals helped write prompts; and Fathom tailored meeting summaries to different users and contexts.

Evaluation helps teams compare choices such as models, prompts, and data selection, and defines what quality means. Habib recommends evolving criteria during prototyping, then using them for production monitoring and regression checks. Teams can combine user actions, issue reports, votes, and edits with focused automated checks and human review; he cautions that broad LLM judging questions can produce noisy results.

Habib advises breaking subjective goals into smaller criteria that can be assessed separately. LLM judges can help with specific checks, such as whether required points appear, alongside code-based measures like precision, recall, and latency. Because quality involves trade-offs among factors such as cost and helpfulness, teams should use a scorecard rather than rely on one overall metric; Hex and Vanta are examples of teams using granular evaluation and human feedback.

Habib highlights three tooling needs: make prompts and evaluations accessible to domain experts, support evaluation throughout development and production, and log inputs and outputs so runs can be replayed. Ironclad built its open-source Rivet library after logging and rerunning traces helped its team debug agents with multiple function calls; Habib says the company reports that about 50% of contracts for its biggest customers are auto-negotiated. Filevine uses Humanloop to manage production prompts with legal experts, data scientists, and product managers.