Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier

New models bring useful capabilities, but adding them to products can introduce unexpected behavior. Thompson calls the work of adapting prompts and addressing regressions the prompt tax, a recurring tension between gaining new capabilities and protecting product reliability.

Orbital automates parts of real estate due diligence, helping lawyers extract information from lengthy property documents and review cited answers. Thompson says the product grew from under one billion tokens per month to nearly 20 billion in 18 months, alongside annual recurring revenue reaching multiple seven figures. The team moved across GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4.1, and reasoning models.

Orbital favored prompts over fine-tuning to ship changes quickly, and used real estate lawyers to contribute domain expertise. Its evaluation process relied mainly on human review and user feedback rather than a comprehensive eval system. For reasoning models, the team found it could remove procedural detail, repetition, and unnecessary constraints; it also used newer models to help migrate prompts.

Thompson recommends feature flags and progressive rollout to manage model upgrades, while recognizing that teams may scrutinize a new model more closely than the old one. Orbital routes product feedback to domain experts, who can adjust prompts and ship fixes in minutes or hours. He argues teams should anticipate how models may improve over the next three to twelve months when making product decisions.

Thompson asks whether human review and fast feedback will keep scaling as agentic products gain more features, or whether evals are needed. He notes that evaluating legal answers involves correctness, style, concision, and citations across many edge cases. His proposed path is to upgrade, roll out gradually, and use incoming feedback to guide expansion.