Behavior Review for AI-Native Delivery | How to Verify AI-Written App Changes

Miho describes how a pull request can look correct and pass CI while the app still behaves unexpectedly on a real device. AI agents make plausible changes faster, increasing review volume while leaving the merge decision dependent on trust. Code review can miss problems that appear only when app components, libraries, and native APIs interact.

The presenters argue that end-to-end tests can be slow, flaky, and poorly matched to the change under review. Their proposed behavior review agent gets a focused mission, such as checking onboarding, checkout, or the flow affected by a pull request. The setup combines CI, an LLM and agent harness, device controls, artifact storage, and a pull request commenting bot.

Miho describes Agent Device, a CLI that lets agents operate iOS and Android simulators, inspect screens, tap and type, and collect logs, metrics, and screenshots. In a demo, an agent checked a new navigation tab on Android and iOS, then reported what it could open and inspect. The screenshots gave reviewers evidence without requiring them to run the app locally.

The presenters recommend deterministic setup, product and platform context, relevant screenshots, and repeatable plans or sample cases. They give an estimated inference cost of a few cents for a simple test, and roughly 50 cents to one dollar for a moderately complex iOS and Android flow including infrastructure. They frame behavior review as an extension of manual QA, which can still catch visual and gesture details that are difficult to specify for an agent.

Miho suggests proving the approach with one workflow and one product area, then expanding if it improves merge decisions. During Q&A, the presenters discuss running agents locally or connecting them to device services such as BrowserStack and AWS Device Farm. They also describe making repeated navigation steps faster and more reliable by saving them as scripts.