A/B testing with React Native

Shetty defines A/B testing as comparing versions of a product to improve customer experience and business metrics. He argues that teams are often wrong about what users want, so product decisions should rely on measured results rather than opinion. Testing can also teach teams about user behavior, even when a variant does not win.

Shetty recommends turning an idea into a hypothesis grounded in a business metric: state what change you expect to affect which measure, and why. Choose one primary metric for the decision, while tracking other measures to understand what happened. In a Booking.com tablet search experiment, 500-pixel images slowed page loads and offset bookings; a later test with 300-pixel images produced a win.

Unlike web changes, native app experiments can take weeks to reach users because of app submission and review delays. Shetty describes fetching experiment settings during app startup, then using an SDK to assign a consistent base or variant to each user and track events. He advises keeping startup responsive and falling back to cached settings or the base version if the network is slow or unavailable.

Changing an entire app at once can show whether results moved without revealing why, while testing only one step of a user journey can improve sign-ups without improving the rest of the experience. Results from another company may not apply: Booking.com users showed little preference between a hamburger icon and a Menu label, while another site reported a 12% lift for Menu. Shetty recommends testing across the funnel and matching experiment size to available traffic.

Shetty recommends testing releases, asking questions when data is unclear, and building a team culture that uses evidence to guide product decisions. Teams with many users can test small changes; teams with fewer users may need larger experiments. He also encourages developers to share what they know through writing and local talks, even when the subject feels familiar.