Giving voice an action in React Native

Wouter describes voice assistants as a growing part of mobile technology, used on phones, in cars, and at home. He outlines a pipeline that converts speech to text, infers a user’s intent, performs an action, and may respond with generated speech.

At Science Mobile Health, Wouter’s team explored voice input and commands for Clean Cards, a medical app that helps patients record vital signs. A demo illustrates the limits of unreliable voice recognition, and he cautions against using voice for critical tasks. He suggests using it to support existing features, such as filling fields or triggering actions.

Wouter demonstrates React Native Voice for speech-to-text, noting that mobile operating systems support more than 40 languages and can recognize speech offline. For interpreting requests, he uses Wit.ai: developers train an app-specific model with example phrases, then send it text and receive likely intents with confidence scores. His example recognizes “Where am I?” as a request to fetch the user’s location.

Wouter explains that iOS Shortcuts can launch an app with data attached, while Google Voice Actions were more restricted at the time. His Siri Shortcut example uses an invocation phrase and user information to open the app and navigate to a relevant screen. He notes that users may need guidance to set shortcuts up.

Wouter shows text-to-speech through Expo modules, where an app passes text and can select a voice and language. He says speech output runs on-device and responds quickly, while the demo also shows that voice interactions can fail. He recommends voice for accessibility and as support for existing app flows.