Artur identifies reliability, privacy, and latency as reasons to run inference on a device. Offline operation avoids dependence on a service provider or network, while local processing keeps data from being sent to a third party. It can also avoid cloud queues and provider downtime.
LLM Inference On-Device in React Native: The Practical Aspects
Speaker
Artur Morys-Magiera, senior React Native engineer at Callstack and PhD candidate at AGH University, discusses practical choices for on-device LLM inference at the React Universe meetup with Zalando.
Models can consume several gigabytes of storage; Artur gives unquantized Llama as an example at roughly five to six gigabytes. OS-provisioned models such as Apple Foundation Models avoid separate app downloads, while GPU and NPU acceleration can improve performance. A case study showed initialization taking about a second on one device but 47 seconds on another, traced to inefficient transpose operations in an OpenCL backend.
Artur compares React Native Fast TFLite, ONNX Runtime React Native, ExecuTorch, and providers in the React Native AI ecosystem. They differ in model formats, hardware backends, platform and OS support, and whether they include tokenization and sampling. React Native AI providers share an interface compatible with the Vercel AI SDK, while Apple Foundation Models require iOS 26 or later for text generation and transcription.
For developers, model compatibility and device capability shape the choice of runtime. Artur recommends considering smaller models for lower-end devices and describes quantization as a way to reduce memory use, with some loss of precision. Compile-time techniques such as operator fusion and memory layout optimization can also improve inference.
The React Native AI providers let developers replace a cloud provider with a local model through a shared AI SDK interface. Custom providers may require downloading and preparing a model first, after which the SDK can handle text generation. Artur also describes OpenTelemetry traces routed through React Native DevTools tooling to inspect generation, streaming, tool calls, and metrics.
Takeaways
- Choose on-device inference when offline access, local data handling, or avoiding cloud latency matters to users.
- Include model downloads and storage in the product decision, especially when users may install multiple apps with separate model copies.
- Benchmark initialization and token generation across representative devices; hardware acceleration can behave very differently across chips.
- Check each runtime's model formats, tokenizer support, platform requirements, and accelerator backends before committing to it.
- Use smaller or quantized models for constrained devices, and trace inference to find performance bottlenecks.
Mentioned in this talk
Search inside this talk
What was claimed 18
- role atArtur Morys-Magiera is senior React Native and AI engineer at
Callstack“I work as a senior React Native developer at Callstack”
0:00inMLC LLM + React Native: On-Device AI Without the PainYouTube Verified4 sources· Talk
- integrates withReact Native AI integrates with Vercel AI SDK
“With all packages under the React Native AI family, we have full support for the AI SDK”
0:18inBuild Smarter Apps: Tool Calling & AI Orchestration ExplainedYouTube Verified4 sources· Talk
- usesReact Native AI uses MLC LLM
“I found an engine called MLC LLM.”
3:51inHow to Run Any LLM On-Device With React NativeYouTube Verified3 sources· Talk
- featureApple LLM: Provisioned with the operating system and available without downloading a separate model
“They come packaged with the OS, so you don’t need to download them”
YouTube Unverified1 source· Talk
- metricMLC LLM: A case study's prefill and initialization took 47 seconds on one device and GPU configuration
“47 seconds were spent on the prefill and initialization phase”
YouTube Unverified1 source· Talk
- featureMLC LLM: Changing the model memory layout fixed inefficient transpose operations on some devices
“If we change the format, it was all good”
YouTube Unverified1 source· Talk
- uses
“The model format it supports is called TFLite. Google has renamed it LiteRT”
YouTube Unverified1 source· Talk
- integrates withReact Native Fast TFLite integrates with Core ML
“It supports Core ML and Metal”
YouTube Unverified1 source· Talk
- uses
“It’s a wrapper for the official ONNX Runtime”
YouTube Unverified1 source· Talk
- featureExecuTorch: Supports Torch .pte models and produces backend-specific files at compilation time
“It supports the Torch .pte model format and optimizes the model at compilation time”
YouTube Unverified1 source· Talk
- featureReact Native AI: Provides a common API for multiple event sources and machine learning runtimes
“bridge almost all the available event sources and machine learning runtimes under one familiar API”
YouTube Unverified1 source· Talk
- builtReact Native AI built Vercel AI SDK
“we’re building it to bridge almost all the available event sources and machine learning runtimes under one familiar API, which you may know from the Vercel AI SDK”
YouTube Unverified1 source· Talk
- usesReact Native AI uses Apple LLM
“It lets you run only Apple Foundation Models”
YouTube Unverified1 source· Talk
- metricApple LLM: Context window: 4,096 tokens
“they have a context window size of 4,096 tokens”
YouTube Unverified1 source· Talk
- featureMLC LLM: Uses compile-time optimizations including operator fusion and memory layout optimizations
“MLC also applies various complex optimizations during compilation, such as operator fusion”
YouTube Unverified1 source· Talk
- usesReact Native AI uses
llama.cpp“a wrapper around llama.rn. It wraps llama.cpp”
YouTube Unverified1 source· Talk
- featureRosanite: Provides customizable and extensible React Native DevTools tabs
“Rosennite provides customizable and extensible React Native DevTools tabs”
YouTube Unverified1 source· Talk
- integrates withReact Native AI integrates with OpenTelemetry
“We created a package called @react-native-ai/dev-tools that provides an OpenTelemetry exporter”
YouTube Unverified1 source· Talk
